rrfss evaluation: issues and strategies
DESCRIPTION
RRFSS Evaluation: Issues and Strategies. RRFSS Workshop, June 19th 2002 Catherine Bingle Sarah Feltis. RRFSS Evaluation Project 2001-2. Project background and development Evaluation objectives and framework Methodology, tools, respondents Results Key issues and strategies. - PowerPoint PPT PresentationTRANSCRIPT
RRFSS Evaluation:RRFSS Evaluation: Issues and StrategiesIssues and Strategies
RRFSS Workshop, June 19th 2002
Catherine Bingle
Sarah Feltis
RRFSS Evaluation Project 2001-2RRFSS Evaluation Project 2001-2
Project background and developmentEvaluation objectives and frameworkMethodology, tools, respondentsResultsKey issues and strategies
Evaluation Project ContributorsEvaluation Project Contributors
RRFSS Evaluation Work Group RRFSS Working Group 2001 Research assistant and project consultant PHRED Funding
BackgroundBackground Evaluation Work Group was formed May 2001 Evaluation framework developed Identified need for support PHRED funding supported hiring of Project
Consultant and Research Assistant
Evaluation ObjectivesEvaluation ObjectivesEvaluate the current state of RRFSSEvaluate the effectiveness of system
processesEvaluate the utility of systemIdentify areas for improvements and
successes of RRFSS
Expectations of the EvaluationExpectations of the Evaluation
Inform changes to RRFSS procedures
Provide a baseline/template for future RRFSS evaluations
Contribute to continuous quality improvement
Evaluation FrameworkEvaluation Framework
Developed by Evaluation Work Group Looked at CDC and WHO surveillance
system frameworks
4 KEY components• Process Issues• Collaboration Among Sites and Key Players• Utility or Usefulness• Cost Effectiveness
Data Collection ToolsData Collection Tools 1st drafts completed in October At least 4 rounds of comment and revision
completed in November after review from the Working Group
In total 5 self- administered questionnaires and one semi-structured interview were developed and used to collect data
Data Collection ToolsData Collection ToolsRRFSS Representative Questionnaire
• Focus– questionnaire development, project administration,
awareness and use of results, data analysis and quality, costs, partnerships, benefits and areas for improvement
MOH Questionnaire• Focus
– familiarity and satisfaction, perceptions of value and impact on health units, feedback on ways to sustain development
Data Collection ToolsData Collection Tools
Program Questionnaire• Focus
– program staff awareness, satisfaction with content of the questionnaire, decision making process, access to results, use of results, impact on programs, and areas for improvements
Non-participating Health Units• Focus
– familiarity, health unit interest in future participation, reasons for not participating, and perceived value of RRFSS
Data Collection ToolsData Collection ToolsISR Interview
• Focus– working relationship between ISR and health units,
questionnaire development process, costs issues and expectations, quality of data and methods, future development of RRFSS
MOHLTC Questionnaire• Focus
– familiarity, impact and value, interest in partnership with RRFSS, strategies for increasing visibility and participation
All 6 tools were disseminated December 2001 All data collected by end of January 2002 109 of 151 (72%) of targeted respondents
participated• RRFSS Representative - 11 of 12 HU’s
• MOH - 10 of 12 HU’s
• Non-participating HU’s - 17 of 24 HU’s
• Program - 69 of 101 program managers/staff
• MOHLTC - 1 of 1
• ISR interview
Data Collection and Data Collection and Response RateResponse Rate
Data AnalysisData Analysis
Quantitative data • Entered and analyzed in SPSS• Frequencies
Qualitative Data• Entered into Word files for content analysis
Process Issues Collaboration Use and Usefulness Cost Issues The Future
ResultsResults
Questionnaire development processes Satisfaction with RRFSS questionnaire Quality of data / methods Data analysis RRFSS dissemination RRFSS awareness
Process IssuesProcess Issues
RRFSS Reps (RR) - RR’s and Program Staff (PS), mostly managers, involved in processes for Q development and/or revision
PS - about 2/5 were involved RR and PS - nearly all very or somewhat satisfied with
processes RRs more satisfied with new module development than with
revision processes
Process IssuesProcess Issues
Questionnaire Development Questionnaire Development ProcessesProcesses
PS concerns: – Q dev. process complexity and length– Need to involve staff more
RR concerns and suggestions:– greater clarity and documentation of policies, responsibilities, procedures, timelines,
decision rationale– greater commitment to process
ISR suggestions (in addition to some of above themes) :– procedures to assure more equality of HU access to opportunities for new questions /
changes
Process IssuesProcess Issues
Questionnaire Development ProcessesQuestionnaire Development ProcessesAreas for ImprovementAreas for Improvement
RR - generally meets HU needsMOHs - split bet. very and somewhat satisfiedPS - more somewhat than very satisfied, esp.
with number of questions relevant to their programs
Process IssuesProcess Issues
Satisfaction with QuestionnaireSatisfaction with Questionnaire
PS and RR - opportunities for more questions - both new topics and expansion of topics
RR - explore rotating coreRR - explore adjusting core : optional balance
Process IssuesProcess Issues
Satisfaction with QuestionnaireSatisfaction with QuestionnaireAreas for ImprovementAreas for Improvement
Most RRs see sample size as adequate Those who do not indicated need for larger sample for
subpopulation analyses Most viewed response rate as good, some excellent.
Likely based on partial info 67% completion rate, June ‘01.
Question quality viewed (ISR) as good overall
Process IssuesProcess Issues
Quality of Data / MethodsQuality of Data / Methods
RR and ISR - further strengthen Q quality procedures - Q review, editing, & testing.
ISR - longer Q testing - pilot study (n=100) where possible (responsiveness important too)
Assessment of existing modules - keeping flawed modules over time problematic
Need feedback mechanism for problems from data user back to Q design Need yearly conference to discuss methods
Process IssuesProcess Issues
Quality of Data / MethodsQuality of Data / MethodsAreas for ImprovementAreas for Improvement
RR - dissatisfaction with timeliness of data receipt at HU
RR - All HU's had begun to analyze RRFSS dataRR - HU's have not enough skilled staff to
analyze data and make it usable
Process IssuesProcess Issues
Data AnalysisData Analysis
ISR, RR - analysis getting short shriftMOH, RR - analytic capacity a key sustainability challenge RR - major staff capacity differences among HU'sKey factor for RR's who were somewhat dissatisfied
overall with RRFSSKey concern of HU's not to participate
Process IssuesProcess Issues
Data AnalysisData Analysis
RR - most said most or all of HU staff aware of RRFSS PS - 2/3 somewhat, 1/3 very familiar with Q content MOH - most very familiar with current status of RRFSS,
most somewhat familiar w/content RR - ½ assess awareness raising processes in HU's as
very effective, ½ somewhat eff.
Process IssuesProcess Issues
RRFSS Awareness in HU’sRRFSS Awareness in HU’s
RR - most HU's had disseminated results in some manner; ½ to broader community
PS - 2/3 had accessed results. Of these: Most had via their epi/analyst staff Most very satisfied w/how results provided 1/3 needed support to use results - all said it was available.
15% weren't sure if needed supports; ½ didn't need.
Process IssuesProcess Issues
RRFSS Dissemination in HU’sRRFSS Dissemination in HU’s
PS - dissatisfaction w/access to results most frequently due to results not being available
PS - need for repeated dissemination PS - need better explanation of data methods
Process IssuesProcess Issues
RRFSS Awareness & RRFSS Awareness & Dissemination in HU’sDissemination in HU’sAreas for ImprovementAreas for Improvement
Most RR very satisfied. Those "somewhat" concerned about cost and staff resources needed for participation
Most PS were satisfied, 42% very satisfied w/how RRFSS is implemented in their health unit
Process IssuesProcess Issues
Overall Satisfaction w/RRFSS Overall Satisfaction w/RRFSS ImplementationImplementation
At non-participating HU's (epi / alt.), ½ very, ½ somewhat familiar w/RRFSS status
Most somewhat familiar w/RRFSS content, some very familiar
PHB rep - very familiar w/status, somewhat familiar w/content
Process IssuesProcess Issues
RRFSS Awareness Among RRFSS Awareness Among Non-ParticipantsNon-Participants
Satisfaction with partnershipContributions to decisions, activitiesPerceived work group effectivenessWays to improve partnership
CollaborationCollaboration
MOH - most very, some somewhat satisfied with how partnership works RR - half very, half somewhat satisfied RR - most say partnership is worth investment ISR very satisfied with effectiveness of working relationship among
partners ISR - benefits of partnership:
– provides ISR w/feedback on data use
– local value of study
– collegial, knowledgeable, committed working group
Collaboration IssuesCollaboration Issues
Satisfaction with PartnershipSatisfaction with Partnership
RR - most feel all HU's have equal voice in decisions, although those involved longer seem to influence decisions more
HU's do not make equal contributions, due to differences in HU staffing, ability to support involvement, choices to be involved
Expectation that these differences will recede as more HU's gain experience
Collaboration IssuesCollaboration Issues
Contributions to Group Contributions to Group Decisions and ActivitiesDecisions and Activities
RR- Working G, Advisory G, and Ad Hoc WG: roughly split 1/2 very, 1/2 somewhat effective
Analysis G and Evaluation G - more somewhat than very effective
ISR perceived as very effective ISR very satisfied with working relationship among partners
Collaboration IssuesCollaboration Issues
Perceived Work Group Perceived Work Group EffectivenessEffectiveness
RR - some duplication between Adv. G and Work. G; difficulty of managing WG meetings
ISR - – Adv. G'd challenge representing larger group– Increasing # of HU's increases coord. difficulty– Range of expertise among HU's – Meeting structures and scheduling– Critical role of Adv. Group - stability must be ensured
Collaboration IssuesCollaboration Issues
Perceived Work Group EffectivenessPerceived Work Group EffectivenessChallenges; Areas for ImprovementChallenges; Areas for Improvement
RR - dedicated funding to Adv. G functions Develop MOU outlining policies, procedures, expectations, roles,
conflict resolution proc. Funding for central coordination of RRFSS Annual face to face meetings Schedule chairs and recorders for WG meets Better documentation of Adv. G and TOR
Collaboration IssuesCollaboration Issues
Ways to Improve PartnershipWays to Improve Partnership
Use of ResultsBenefits to Health Units from RRFSSImpact on Program Planning & EvaluationImpact of Not Having RRFSSImportance for Public Health SurveillanceNeeds Not Met By RRFSS
Use and UsefulnessUse and Usefulness
Virtually all HU's had used RRFSS dataUse highest among epi's and program staff. Managers,
media, and MOH also cited.Use highest among CDP programs. Use also frequently cited for Env. HP & VPD programs;
research & communications staff
Use IssuesUse Issues
Use of Results - RRUse of Results - RR
38% used results in their programsOf those who had not, ½ said this was due to results not
yet being availableRemainder said they had not yet had timeSome indicated data not of sufficient importance /
relevance to use
Use IssuesUse Issues
Use of Results - PSUse of Results - PS
Barriers to use:– time required for data analysis and administration– awareness among PS
Key needs:– funding for central project coordination– common syntax files – better dissemination efforts in HUs
Use IssuesUse Issues
Use of Results - RRUse of Results - RRBarriers; Areas for ImprovementBarriers; Areas for Improvement
Current, timely, local data, filling data gapsData for emerging and locally relevant issuesStaff use for planning and evaluationBetter quality than typical HU surveysStrengthened partnership among HUs and epidemiologists, and
w/community partners Increased understanding of surveillance
Use IssuesUse Issues
Key Benefits from RRFSS - RRKey Benefits from RRFSS - RR
All MOH -- RRFSS results will have impact on HU ability to meet PP&E standards
8 RR agreed, 3 "too early to say"44% of PS agreed; 26% did not; 30% not sureNon-part. HU's - 14 agreed; 3 "too early"PHB rep - agreed
Use IssuesUse Issues
Expected Impact on Program Expected Impact on Program Planning and EvaluationPlanning and Evaluation
MOH, RR, & PS identified same range of negative impacts on PH effectiveness:
Reduced access to local, timely, adaptable, continuous data Reduced ability to measure program objectives, reduced
effectiveness in PP&E, and overall effectiveness / accountability Need for additional investment in HU- and topic-specific surveys
Use IssuesUse Issues
Impact of Not Having RRFSSImpact of Not Having RRFSS
Majority (8/11) of RRFSS Reps were “very satisfied” w/RRFSS as surveillance method
Majority (8/10) of the MOH’s from participating health HU’s view RRFSS as a “very important” tool for public health surveillance
11 of 17 non-participating HU’s also view as a “very important”
Use IssuesUse Issues
Importance for Public Health Importance for Public Health SurveillanceSurveillance
Virtually all HU’s indicated the need for other community health survey work beyond RRFSS
• Child and youth health assessment• Surveying of sub-populations e.g. pregnant women• HU specific topics• Sensitive topics• Information not obtained through telephone survey
Use IssuesUse Issues
Needs not Met by RRFSSNeeds not Met by RRFSS
Key Staff Involvement in RRFSSReturn on InvestmentCost Reduction Strategies
Costs and Cost-Effectiveness Costs and Cost-Effectiveness
Time• Annual estimated key staff = 0.68 FTE per HU (range 0.16 to 2.0 FTE per HU)
WHO• Epidemiologist• Other research staff
Activities• New module dev., data analysis, and RRFSS WG meetings• Data dissemination and presentation of data to HU staff
Cost and Cost Cost and Cost EffectivenessEffectiveness
Staff Involvement in RRFSSStaff Involvement in RRFSS
RRFSS Reps - (5/11) rated as “good” and 3 rated as “excellent” (time, money, and resources)
MOH - divided between “very satisfied” and “somewhat satisfied” with their HU’s investment in RRFSS
Program Savings - most RRFSS Reps indicated “to early to say” whether RRFSS had resulted in any program saving or would create revenues for HU in future
Cost and Cost Cost and Cost EffectivenessEffectiveness
Return on InvestmentReturn on Investment
Support for central coordination of RRFSS
Encourage partnerships with other organizations
Over time - improvements in organization and efficiency
Cost and Cost Cost and Cost EffectivenessEffectiveness
Cost Reduction StrategiesCost Reduction Strategies
Sustainability Challenges Expanded HU and Provincial ParticipationStrategies for Increasing ParticipationSuggestions for Raising RRFSS Visibility
The FutureThe Future
• MOH Perspective• Adequate staff/resources • Rapid analysis of data• Funding/Cost• Keeping program staff involved
• RRFSS Rep’s Perspective• Cost/Funding • RRFSS administrative work• Questionnaire content
The FutureThe Future
Sustainability ChallengesSustainability Challenges
• ISR Perspective
• Whether or not able to do all 38 HU
• Need signed commitment from HU earlier, and for longer term
• Working group needs support for methodology improvements and analysis
The FutureThe Future
Sustainability ChallengesSustainability Challenges
• HU not currently participating in RRFSS, over half indicated an interest in participating in the future
• 6 reportedly were considering • 4 were actively preparing
• HU not interested or not considering participating most common reason given for this was COST
• lack of staff• time
The FutureThe Future
Expanding HU and Provincial Expanding HU and Provincial ParticipationParticipation
• RRFSS Rep’s and MOH were asked to agree or disagree with following statements
• Important for all health units to participate• Important for MOHLTC to participate• There should be a province wide sample• Funding for RRFSS should be on 100% provincial basis
The FutureThe Future
Expanding HU and Provincial Expanding HU and Provincial ParticipationParticipation
• RRFSS Reps Perspective• Agreement strongest for important for MOHLTC to participate and there should be a
province wide sample • 6 Strongly, 2 agree important for all HU to participate (3 disagree)• 2 strongly, 5 agree funding 100% provincial basis (4 disagree)
• MOH Perspective• Agreement strongest important for all HU to participate • Most agreed important for MOHLTC participate and funding 100% provincial basis• Agreement was least strong there should be province wide sample
The FutureThe Future
Expanding HU and Provincial Expanding HU and Provincial ParticipationParticipation
• Ministry agrees to cost share• Provision of core questionnaire funding• Bring attention to non-participating HU• Benefits realized, participation should increase• Culture of evaluation in HU• Use of RRFSS data
The FutureThe Future
Strategies for Increasing Strategies for Increasing ParticipationParticipation
• Publish/present data• Share results and information • Target dissemination • Link to academic centres• Provide key results to PHB to support internal government processes• HU provide key results to local politicians
The FutureThe Future
Strategies for Raising VisibilityStrategies for Raising Visibility
ConclusionsConclusionsWhat conclusions should be drawn from the
results, and what are the Next Steps?To be determined … in part by our
discussions todayWhat strategic issues are raised?How will they be addressed?
Issues, Strategies, and Issues, Strategies, and Next StepsNext Steps
Adequate staff/resources for analysis and making data useful
Increasing use of data in program planning and evaluation Dissatisfaction with getting question accepted on
questionnaire (limited # of questions) Increasing sample size Process for revising old and developing new modules RRFSS coordinators overwhelmed by work
Issues, Strategies, and Issues, Strategies, and Next StepsNext Steps
Need for stability and support for central work of RRFSS partnership Unequal HU access to ISR, decision-making, workload and resources Upgrading RRFSS web site Sustaining funding for existing participants and new funding for
interested ones Provincial representation of data Sharing data among organizations Improved timeliness of data ISR ability to handle increase