conciliating scientific rigor & pragmatics in outcome evaluation research for human service...

Conciliating scientific rigor & pragmatics in Outcome Evaluation Research for human

service professionalsTheoretical underpinnings and implementation techniques

Network for Health & Welfare Studies

Dr. Charles C. ChanDr. Amy P. Y. HoMr. Kevin Chan

February 25, 2004

ConvenorNetwork for Health & Welfare Studies

Associate ProfessorDepartment of Applied Social Sciences

Dr. Charles C. Chan

Conciliating scientific rigor & pragmatics in OER

Preamble

• Applied research can be of better service to the human service industry, especially in a quality conscious era, if it promises to pay balanced attention to the dialectical relationship between the professional / researcher and the service recipient.


Quality and Effectiveness :

• Strange bed-fellows or estranged couples?


• The overall all goal of placing priority on both quality and effectiveness of health programs is to document the synergistic contribution of its component parts to a holistic improvement in health and welfare of the population.

WHO-EURO Working Group on Health Promotion Evaluation (1998)



• The focus should not be on documenting quality or evaluating effectiveness of isolated interventions as end points, but rather on the relationship of a given intervention to the other components of the health promotion strategy. Such an analysis may indeed provide a fresh approach to the very issue of quality.



• BARD is a tested approach in applied research to address the demands of scholarship of application and community benefits

• It recognizes problems central to applied research

• It contributes by directing decision making in the research process

Definition of the Balanced Applied Research Development Model (BARD):


1. Placing value on a balanced approach to the scientific requirements and the pragmatics of outcome evaluation in human services

Main features of BARD:


2. Emphasizing integrating research into practice by transferring skills in data capturing to professionals or lay persons in service delivery



3. Analyzing as much as possible, the triadic relationship between the professional, lay person, and service recipient and sustaining intervention effects through supportive supervision



Efficacy Vs Effectiveness

• Efficacy – Intervention does more good than harm under optimum conditions

• Effectiveness - Intervention does more good than harm under real-world conditions

(Flay, 1986)


The RE-AIM framework (Glasgow, 1999)

• Reach

• Efficacy or Effectiveness

• Adoption

• Implementation

• Maintenance and cost


RE-AIM issue Efficacy studies Effectiveness studies

Reach Homogenous Heterogeneous

Efficacy or Effectiveness Very specialized & standardized; randomized

Broad intervention; not necessarily standardized nor randomized

Adoption Run in controlled experimental setting

Run in real-world setting with little control

Implementation Run rigorously by well-trained research staff

Run by staffs in service setting with competing demands

Maintenance and cost Less interest on maintenance and cost

Major concerns in real settings

The RE-AIM framework (Glasgow, 1999) - Cont’d


Outcome Evaluation Research in Hong Kong – Financial Landscape

• Health Services Research Fund (1995 - 2003)

• Health and Health Services Research Fund (2003 - present)

• Research Fund on Control for Infectious Diseases (2003 – present)

• Community Investment & Inclusion Fund (2003 - present)


• Capital amount of $55M

• Total number of submission: 1096

• Total number of approved HSRC projects=224

Health and Services Research Fund (1995-2003)


Health and Health Services Research Fund (2003 - present )

•Capital amount of $10M

•Total number of submission: 187

•Total number of approved projects=4 (ESGAA meeting in July 2003)


Research Fund on Control for Infectious Diseases (2003 – present)

• Capital amount of $ 500M (50M given to PRC’s Ministry of Health for SARS research)

• Open call and commissioned research activities: tentative 30M to HKU in basic and epidemiological modeling research & 25 M to CUHK on public health and emerging infectious diseases drug and treatment development research

• Leaving sums for the establishment of the Centre for Health Protection


Community Investment & Inclusion Fund - Funding information

Project No. Name of Project Name of Applicant OrganizationGrant Approved

0015-01 Helping Others - Self Growing ProjectChristian Concern for the Homeless Association $579,600

0023-01Project to Promote Family Cohesion & More Colourful Family Life

Pok Oi Hospital Madam Chu Kok King Child Care Centre $189,630

0069-01Collaborating People in Tsuen Wan Rural Areas The Yuen Yuen Institute $750,910

0070-01'Healthy Mothers - to-be' - A Women and Community Networking Project

Kwun Tong Methodist Social Service Grassroots Occupational Service Unit ' Healthy Mothers - to-be' Club $232,040

0079-01 Sunny Community Program

Hong Kong Outlying Islands Women's Association, Jockey Club Women's Integrated Service Centre $1,045,800

0091-01It's All Because of You: Community Cohesion Project Richmond Fellowship of Hong Kong $209,160


Community Investment & Inclusion Fund - Funding information -Cont’d

Project No. Name of Project Name of Applicant OrganizationGrant Approved

0092-01Space of Dream - Youth Employment Project in the North District

The Evangelical Lutheran Church of Hong Kong, North District Integrated Youth Service Centre $1,107,700

0100/0101-01 "Caring Estates" in Southern District

HK Southern District Women's Association Ltd and Aberdeen Kaifong Welfare Association Social Service Centre $2,042,800

0113-01Community Care & Networking Programme for "Po Tin" Residents Urban Peacemaker Evangelistic Fellowship $1,005,300

0127-01Cross Generational Community Integration Programme

SAGE Eastern District Multi-service Centre for the Elderly $264,600

0133/0134-01

Retired Senior Volunteers' Association - Community Mutual Help Network Project St James' Settlement $954,740

0151-01Family Service Ambassador Scheme, Caritas Family Service Caritas - Hong Kong Family Service $599,700

Total $8,981,980

Dr. Amy P. Y. Ho

Member

Network for Health & Welfare Studies

Senior Lecturer

Department of Applied Social Sciences


OER in human service setting – An example

• “Promoting health and well-being of elderly patients with chronic illness: A coordinated medical and social service program”

• Funded by the Health Care & Promotion Fund


Recruitment of participants in participating hospital and obtain their consent to join the programme

Experimental group will receive 1) Comprehensive Discharge Planning and need assessment by Medical

Officers of participating hospital and Social Workers of NGO 2) Baseline Assessment and re-assessment at 3 months interval by

researcher of HKPU 3) Coordinated package of medical and social services provided by the

project

Participating NGO

-Provide tailor-made psychosocial

service using case management

approach

Regular Meetings and Case-conferences to

exchange information on participants

Participating hospital -Provide medical care

Case Manager to

coordinate all

social services

Home-based

services

Monthly home

visit by trained

volunteers

Community-based

services

Medical follow up at SOPD by the same Consultant Geriatrician & referrals to other medical services

Final Assessment / Evaluation of Project Effectiveness by The Hong Kong Polytechnic University

Randomize the participants into experimental group and control group

Control group will receive

1) Baseline assessment and subsequent 3 months interval

re-assessment by researcher of HKPU

2) Routine medical and social services offered by HA &

NGOs in the community

Hong Kong Polytechnic

University

- Data collection and data

analysis

Referrals to

other social

services

Intervention flowchart


Standardized Interventions Intervention Potent elements Indicators for treatment integrity 1. Comprehensive hospital

discharge planning (carry out by both the medical officers and social workers)

To ensure that the participants receive coordinated and comprehensive medical and social services

Home visit to the participants should be carried out within 2 weeks after discharge

Both the hospital and welfare agency have the same set of patient records

2. Special outpatient clinic service

Participants are seen by the same consultant geriatrican & the participant is accompanied by the same social worker to attend appointments

Progress reports of each participants are keep by the case-mangers (filled out by service providers of the welfare agency)

The consultant geriatricans will inform the case manager of any changes in medical services to the participants

3. Monthly home visit Carry out by a group of relatively stable, trained volunteers to ensure continuity and quality of service

Standardized ongoing training is provided to the volunteers (with training manuals)

Home visit report is submitted to the case manager after each visit

Bi-monthly meeting will be held to maintain the motivation and commitment of the volunteers

4. Progress monitoring by the same case manager

The progress and needs of the participants are closely monitored by the same case manager

The case manager will call monthly case conference (attended by the consultant geriatricans) to discuss the progress of the participants.

Individualized Interventions Additional, tailor-made medical and social services are provided by the hospital and the welfare agency

To provide additional support to participants who have changing needs

The additional services are recorded and participants’ situations are closely monitored

Intervention components


Issues in unblinded study

• Feasibility of blinding in human service research

• Artifacts introduced by confounders in unblinded study


Standardization of intervention

• Contrasting the notion of “experimental group” in the context of medical research and human services research

• “Black Box” phenomenon in health promotion research (Moore, 2003)


Control on extraneous variables

• Between-group imbalance in the control of extraneous variables

• Incorporating extraneous variables in the analysis


Data quality and reliability

• Staff turnover

• Artifact due to unblinded group assignment

• Reliability of self-reported outcomes


Issues in data analysis

• Loss to follow-up

• Causes of participants attrition in human services OER

• Effect of attrition on data analysis

Mr. Kevin Chan

Research AssociateNetwork for Health & Welfare StudiesDepartment of Applied Social Sciences

Honorary Scientific OfficerThe Hong Kong Childhood Injury Prevention & Research Association


Presumption of the experimentalist ideal (Smith, 1985)

• OER in human services are more than often violating the assumptions required in traditional RCT convention

• Such difference should not be viewed as a deviation from the conventional approach, but rather a calling for applied research with stronger ecological validity


In fulfilling requirements set for RCT, we often undermine the following (Rolls, 1999):

• The difficulties associated with the random assignment• The ethical and administrative objections to

randomization (de_Raeve, 1994)• That inputs are rarely stable• That awareness of difference introduces bias• That it is almost impossible to exclude extraneous

variables• The research is unable to say why changes detected

have occurred (Newell, 1992)


Troubleshooting to barriers – OER in human service setting

• Dealing with less-than-perfect random assignment• Adjusting for unstable input in human services OER• Statistical control for unblinded studies• The inclusion of extraneous variables as covariates• Narrowing down to a closer approximation of causal

relationship


Troubleshooting to barriers – Group assignment in OER for human service setting

• Balancing incentives for follow-up of control group• Intention-to-treat analysis• As-treated analysis• Complier-Average Causal Effect (CACE) analysis


Troubleshooting to barriers – Group assignment in OER for human service setting – Cont’d

Balancing incentives for follow-up of control group

• Despite the lack of intervention, control group participants could be prescribed with “non-specific” intervention or substantial support to maintain their interests to comply with the data collection process



Balancing incentives for follow-up of control group

“Non-specific” intervention

-E.g. Delivery of printed health promotion materials

Substantial support

-E.g. Safety device, health supplements



Intention-to-treat analysis As-treated analysis

•Treating non-compliers / drop-outs with compliers as same

•Adhere to original group assignment but lead to bias estimate of treatment effect (i.e. magnitude of intervention “dragged down” by non-compliers / drop-outs

•Realigning participants group assignments according to actual intervention received or sought

•Used as a secondary analysis to contrast the IIT results

•Subject to selection bias in case of noncompliance (i.e. those longing for an integrated medico-social care package might be more active in maintaining physical and mental health)



Complier-Average Causal Effect (CACE) analysis

(Angrist, 1996; Little, 1998)

• Expanding the conventional regression equation with a new term – compliance with intervention




(Angrist, 1996; Little, 1998)• Difference of evaluation outcome (e.g. score on SF-36,

probability of sustaining home injury) between compliers and non-compliers, with other missing information (e.g. covariate variables such as age, gender, health status, attitude toward the research question) adjusted and stratified by compliance category.




(Angrist, 1996; Little, 1998)• CACE accounts for potential interaction between

randomized group assignment & compliance and allows more accurate estimate of program effectiveness by:

i. Increasing number of “usable” cases

ii. “Subtract” the artifact of compliance that moderates the outcome variable


Troubleshooting to barriers – Adjusting for unstable input in human services OER

• Explicit definition of intervention protocol

i. Behavioral interventionii. Substantial inputiii. Personnel inputiv. Temporal data (Date, time, duration, frequency)v. Define boundary of intervention


Troubleshooting to barriers – Adjusting for unstable input in human services OER


Troubleshooting to barriers – Adjusting for unstable input in human services OER – Cont’d

Process analysis

• Process evaluation fills up the “Black Box” (Moore, 2003) left void in health promotion research and strength internal validity of the intervention under investigation.



Process analysis• Process analysis identifies variance in protocol

implementation & participants’ receptiveness to the prescribed intervention. A process analysis covers

i. Count of activities attendedii. Perceived effectiveness of the interventioniii. Compliance to intervention protocoliv. Met and unmet needs related to the intervention



Process analysis (Cont’d)• The information gathered from the process research

should be integrated into the outcome research, rather than setting aside as auxiliary document in the discussion of study results


Troubleshooting to barriers – Quality Assurance in OER for human service setting

Quality assurance exercise

Organizational requirement

Staff involvement


Maintenance and cost in OER

• Unanticipated effect of health promotion on social capital and cohesion (Raphael, 2000)

• Economic analysis (including Cost-effectiveness Analysis or Cost Benefit Analysis)

Conclusion


Decision questions generated from the BARD Model

Q1. How can the range of problems be considered as part of public service (health, welfare, and education) priority?



Q2. What is the status of the efficacy studies on similar intervention programs in the international literature?



Q3. Who should do what and how much for whom before one can consider the intervention effective?



Q4. Should we vary the intensity of intervention (e.g. tiered multi-level program) to suit targeted service recipients?



Q5. How can staff training and documentation of quality manual help differentiate role and skills for professional and lay participation to ensure intervention integrity?



Q6. Is the technology transferred in the process of conducting OER sufficient to ensure human and financial sustainability?


WHO-EURO Working Group on Health Promotion Evaluation (1998) pointed out this :

Evaluation help them find out what works and how; make decisions about future health promotion policy strategies and programs,….modify and improve current programs; justify policy choices; and increase impact and effectiveness (including cost-effectiveness) of health promotion initiatives.

Some final words for the Policy-makers…


Major NHWS OER publicationsChan, C. C. (2000). Workbook on measurement & use of

outcomes in social services. Hong Kong: Hong Kong Polytechnic University.

Chan, C. C., Chow, C. B., & Chan, K. (2003). SARS and social cohesion in Hong Kong: A research agenda. Asian Pacific Journal of Social Work, manuscript in review.

Chan, C. C., Lui, W. S., Wan, D., & Yau, S. (2002). Evaluating Service Recipient Outcomes in Psychiatric Residential Services in Hong Kong. Research on Social Work Practice, 12(4), 570-581.

Chan, C. C., Lui, W. S., Wan, D., & Yau, S. (2002). A multilevel approach to supported employment outcomes in Schizophrenia. American Journal of Orthopsychiatry, manuscript submitted for publication.


Major NHWS OER publicationsChan, C. C., Luis, B. P., Chow, C., Cheng, J. C. Y., Wong,

T. W., Chan, K., & Tsui, S. (2001). Validating narrative data on residential child injury. Journal of Safety Research, 37(4), 377-389.

Chan, C. C., Luis, B. P., Chow, C. B., Cheng, J. C. Y., Wong, T. W., & Chan, K. (2003). Unintentional residential child injury surveillance in Hong Kong. Journal of Paediatrics and Child Health, 39(6), 420-426.

Chan, C. C., Tsui, M. S., Chan, M. Y. C., & Hong, J. H. (2002). Applying the Structure of the Observed Learning Outcomes (SOLO) Taxonomy on Student's Learning Outcomes: An empirical study. Assessment & Evaluation in Higher Education, 27(6), 511-528.


Major NHWS OER publicationsChan, C. C., Wan, D., & Lui, W. S. (2002). Determining

performance standards for supported employment service in Hong Kong. Hong Kong Journal of Social Work, 36(1 & 2), 95-111.

Cheung, F., Yau, A., & Wong, K. (2001, February 22, 2001). The Costs of Cost-Effectiveness: An initial review of the competitive bidding experience in contracting out home care service. Paper presented at the Seminar on Efficiency & Effectiveness of Social Care, Part III, The Hong Kong Polytechnic University, Hong Kong.

Chow, C. B., Cheung, W. L., & Chan, C. C. (2002, April 28-30, 2002). Transforming AEIS data for the prevention of unintentional childhood injuries in Hong Kong. Paper presented at the Hospital Authority Convention 2002, Hong Kong.


Major NHWS OER publicationsHo, A. P. Y. (2001, May 19, 2001). Creating a supportive

environment for elderly with chronic illness. Paper presented at the Healthy Ageing Convention, Hong Kong, Hong Kong Convention and Exhibition Centre.

Chan, C. C., & Chan, K. (2003). Competence-based evaluation of volunteer progress. Evaluation and Program Planning, manuscript in review.

Network for Health & Welfare Studies, the Hong Kong Childhood Injury Prevention & Research Association. (2003). The Volunteer Home Visitation Programme – Training Manual

Network for Health & Welfare Studies, the Hong Kong Childhood Injury Prevention & Research Association. (2003). The Volunteer Home Visitation Programme – Quality Assurance Manual


Key ReferencesAngrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of

Causal Effects Using Instrumental Variables. Journal of the American Statistical Association, 91(434), 444-455.

Flay, B. R. (1986). Efficacy and effectiveness trials (and other phases of research) in the development of health promotion programs. Preventive Medicine, 15(5), 451-474.

Glasgow, R. E., Lichtenstein, E., & Marcus, A. C. (2003). Why Don't We See More Translation of Health Promotion Research to Practice? Rethinking the Efficacy-to-Effectiveness Transition. American Journal of Public Health, 93(8), 1261-1267.

Little, R. J., & Yau, L. H. Y. (1998). Statistical techniques for analyzing data from prevention trials: Treatment of no-shows using Rubin's causal model. Psychological Methods, 3(2), 147-159.

McPherson, K., & Britton, A. (2001). Preferences and understanding their effects on health. Quality & Safety in Health Care, 10(Supplement 1), 61i-66.

Moore, L., Graham, A., & Diamond, I. (2003). On the feasibility of conducting randomised trials in education: case study of a sex education intervention. British Educational Research Journal, 29(5), 673-689.


Key References

Newell, D. J. (1992). Randomised controlled trials in health care research. In J. Daly & I. G. McDonald & E. Willis (Eds.), Researching health care : designs, dilemmas, disciplines (pp. 47-61). New York: Routledge.

Raphael, D. (2000). The question of evidence in health promotion. Health Promotion International, 15(4), 355-367.

Rolls, L. (1999). The Challenge of Evidence-based Practice. In E. R. Perkins & I. Simnett & L. Wright (Eds.), Evidence-based health promotion (pp. 47-50). Chichester, New York: John Wiley.

Smith, G. (1985). Assessing health care : a study in organisational evaluation / Gilbert Smith and Caroline Cantley. Milton Keynes: Open University Press.

Thompson, B., Coronado, G., Snipes, S. A., & Puschel, K. (2003). Methodologic advances and ongoing challenges in designing community-based health promotion programs. Annual Review of Public Health, 24(1), 315-340.

The End!

Thank you for coming.

For further informationVisit us at

http://www.acad.polyu.edu.hk/~ssnhws/en/index.html

conciliating scientific rigor & pragmatics in outcome evaluation research for human service...

Documents

oer quality

oer bard

chan slide

scientific requirements

cost slide

oer preamble

outcome evaluation research

health welfare studies