proposal and program evaluation ivan yau. critical reflections different purposes / targets lead to...
TRANSCRIPT
Critical Reflections
Different Purposes / Targets lead to different Strategies and Demands on rigorousness of program evaluation – to public, visual documentation and stories can be used effectively (have a sense that these are representative), to academic and funder, generally, statistical analysis is used (but should have meaningful no.!!)
Critical Reflections
Poor in Program Evaluation in general so far!! – these can be reflected in program evaluation published with poor objectives, evaluation methods and its interpretation. So what are your roles as social workers?
Critical Reflections Evaluation on Program Effectiveness
are linked to ‘accountability’ and ‘monitoring’ both. If impact / effectiveness can be demonstrated (accountability), our service can only be monitored by output.
Program Evaluation is a progressive and goal-driven process, not a repetitive routine measure, e.g. after a rigorous evaluation on effectiveness, only certain kind of QC or a more in-depth evaluation is required.
Some examples
1. Hong Kong Family Welfare Society (1989) A research report on cognitive-behavioral Group Therapy. - further developed into more refined program on clients with more severe M.I. symptoms, parent group and other with more skill deficits
2. Chronic Disease Self Management Program (CDSMP) – more vigorous screening is required to ensure the commitment in the program and reduce drop-out rate
Critical Reflections Evidence – biased preference1.Numbers – even converting subjective
experience into Numbers!!
2.Counting and Rating3.Statistical Analysis (T-test, sign-test,
…logic of significant level and hypothesis testing)
Program Logic Two Holistic View: Analysis of Target and Problem, Mechanism of
Change (Intervention), Objectives, Evidence and Expected changes
Program Evaluation Objectives:1.Change2. Important3.From SW concern / perspective!! Effective? - Evidence (changes
observed) is consistent with the objective.
Illustration – Program Logic
跨越鴻溝 – 促進親子溝通計劃( 2001) 針對大埔區青少年不斷離家出走的現象 合作伙伴:警方(轉介) 潛伏危機:受誘惑而去吸毒、販毒、賣淫 服務目標所受的影響:減少再次離家出走的情況 社會控制的議程:加派警力巡查、對離家出走的青少年 /家長提出警告、向多次離家出走青少年的家庭提出保護令的申請、……
Illustration – Program Logic 現象分析:親子缺乏溝通是一般離家出走少年的起源?
理論:青少年成長理論、家庭發展歷程、…… 進一步探索:1. 青少年離家後在甚麼地方流連?有何活動?跟甚麼人連繫?(滿足甚麼需要?)
2. 這些青少年的家庭狀況?他們對家庭的觀感?家庭關係?(家庭無法滿足他們甚麼需要?)
Mechanism of Change Some Illustrations
1. Skill training – Parenting skill training as an example structured exercise, brief lecture, demonstration, role-play and feedback, experimentation in real life
2. Cognitive restructuring – getting familiar with ABC (differentiating DT /DT, aware of undesirable impact of DT), identifying own DT, disputing these through empirical enquiry, self-restructuring process
Mechanism of Change Some Illustrations3. Self Understanding / acceptance – self-
exploration on critical incidents in trustful environment, feedback from others after intensive interaction, connecting with others deeply and comparing the difference
4. 生死教育 – reflection on the edge of life / death experience or stimulated experience
Some Clarifications Intervention is different from
Objective / Effectiveness. Process review is useful for
improving the practice. Client’s satisfaction is Not the
program effectiveness.
Objective Usually viewed as ‘Prevention of
Undesirable Consequence’ / ‘Alleviation of the (social) Problem’
Important and Desirable – from who’s perspective? Client’s?
Influence from stake-holder? Theory-driven? Social Control?
Effectiveness and Change Extent: Magnitude in relation to all
participants (sometimes Magnitude alone is sufficient!)
Confidence: Change is not the result of chance
Causal relationship: can hardly be established
Intervention Intervention must link to mechanism
of changes (theory-driven and common sense!!) and desirable outcome.
Group proposal – brief session outline first, and further detailed session plan as the group evolves gradually.
Effectiveness I
Participants’ change (pre-post, post only) Issues of measurement: validity (face
validity), reliability (items-correlation) and sensitivity
Changes reach statistically significant level Single-item comparison Scale construction: literature review, local
validation, feedback from participants,…
Effectiveness II
Outcome Indicator Criteria for success: how come? Baseline makes comparison feasible
(indicating the change) Meaningful (and significant) to reflect the
accumulated effort, or indirect evidence to achieve the ultimate goal (community level intervention)
Different Levels – G & P Group and Programs1. Specific target group2. Focused, limited intervention3. Operationalization of Objectives –
pre-post measures, post-measures4. Use of statistical analysis to assess
significant changes
Different Levels - Project Project1. Specific target group with rather
long-term intervention2. Focus on several key outcome
indicators3. Pre-post measures are also feasible
Different Levels - Service Service1.Diversified target groups with long-
term intervention and general / vague objectives (IFSC, DEC,…)
2.Aggregated data is always useful3.Focused on major groups, or use of
randomizations if necessary
Political Agenda Making use of client’s views in
advocating for changes, e.g. IFSC, CSSA, SW field work supervision, 自力更新服務
Connecting impact of the program with advocacy / policy change, e.g. 類風濕性關節炎自我管理課程、中學義工服務、小學學校社工服務、兒童之家服務、…
Rounding Up Aggregate data (client’s satisfaction, objective reached,
target complaints, etc) is useful for service evaluation, key outcome indicator (decrease in A & E of child injury due to child neglect, decrease in suicide) for project evaluation, while accumulated impact (make use of community resources by NA, support network, no. of high risk case receiving service) for both.
Rounding Up Evaluation should be better linked to
continuous improvement (don’t repeat the
evaluation every year), service development (promotion of a better service model) or social movement (advocacy) (e.g. changing concepts of chronically ill patient – ownership of health, instead of replying on medical treatment).
Rounding Up Evaluation can be and should be Political
– particularly help marginalized group to voice out their views / needs.
Social workers should:1. Be able to link up the objective,
intervention and outcome.2. Be quipped with themselves of different
ways / strategies for program evaluation.