Draft Outline of Draft Outline of

Framework for Evaluation Framework for Evaluation

of Family Resource Centresof Family Resource Centres

Kieran McKeownKieran McKeown

Version V6 – Version V6 – 77thth September September 20 2011, CDI Conference in Dublin11, CDI Conference in Dublin

Draft Outline of Draft Outline of

Framework for Evaluation Framework for Evaluation

of Family Resource Centresof Family Resource Centres

Kieran McKeownKieran McKeown

Version V6 – Version V6 – 77thth September September 20 2011, CDI Conference in Dublin11, CDI Conference in Dublin

Family Support AgencyFamily Support AgencyFamily Support AgencyFamily Support Agency

Evaluation QuestionsEvaluation QuestionsEvaluation QuestionsEvaluation Questions

1. The first set of evaluation questions are:

To what extent is the FRC programme achieving its objectives, that is, one or more of its seven outcomes?

Would these outcomes, for individuals and/or communities, have been achieved without the programme?

In order to answer these questions, it is necessary to make a before-and-after comparison of two identical groups: those who participate in the programme (the ‘treatment group’) and those who do not (the ‘comparison group’).

A more tentative answer could be provided by comparing how FRC areas have changed relative to national norms for each of outcome

2. The second set of evaluation questions are:

For any given FRC outcome, what are the factors or ‘pathways’ influencing its effectiveness, for whom, in which group of FRCs, using what type of intervention(s), etc?

In order to answer these questions, it is necessary to have a theory about how the FRC intervention works and to test this against the evidence. This requires a ‘logic model’ for each programme outcome that is being evaluated.

Draft Logic Model of FRC ProgrammeDraft Logic Model of FRC ProgrammeDraft Logic Model of FRC ProgrammeDraft Logic Model of FRC Programme

Inputs Pathways Outcomes

Changes in Behaviour & Attitudes of Parents & Children:- Individual well-being- Relationships- Support networks- Neighbourhood (trust, safety)

Changes in FRC & Community Capacity:- Staff skills, attitudes, practices - Engagement & ‘empowerment’ of community

Interventions with parents, children & community:- Type- Duration- Intensity

Uptake and impact of interventions may be influenced by characteristics of:- Target group - Target area- Outcome area

Improved Child & Parent Outcomes:1. Healthy / healthier2. Actively learning3. Safe from harm4. Economic security5. Safe environment6. Positive networks7. Participating in society

Changes in Local & National Service Capacity: - Response of other services- Inter-agency cooperation- National policy

Comments on Comments on Draft Logic Model of FRC ProgrammeDraft Logic Model of FRC ProgrammeComments on Comments on Draft Logic Model of FRC ProgrammeDraft Logic Model of FRC Programme

This model is the first draft of a theory on how FRCs work. It is only the first step and requires further refinement within the ‘learning community’ of the FRC programme.

It is essential that the evaluation is developed in tandem with the roll-out of the framework in those FRCs who participate in the evaluation, focusing on a small number of substantial and sustained interventions.

This logic model also provides a simple framework for writing up an illustrative case study of an intervention.

This approach involves the FRC programme preparing itself for evaluation by specifying exactly what outcomes it is trying to achieve, ensuring that they are achievable and measurable, deciding on the pathways by which they are to be achieved, with whom, and over what time period, etc.

This is already part of the task of preparing a local family support strategy, but could now be done by FRCs in the knowledge that their local strategy will be evaluated.

In this sense, developing the evaluation framework could be seen as an integral part of the FRC strategy and building FRC capacity.

Further Comments on Further Comments on Draft Logic Model of FRC ProgrammeDraft Logic Model of FRC ProgrammeFurther Comments on Further Comments on Draft Logic Model of FRC ProgrammeDraft Logic Model of FRC Programme

From an evaluation perspective, each ‘box’ in the model represents data to be collected; from an implementation perspective, each box represents a series of actions to be considered and delivered.

‘Inputs’ are ‘FRC interventions’, both ‘programmed’ and ‘other developmental’ activities. ‘Inputs’ also include indicators of need and the factors that may influence participation in FRC activities.

‘Pathways’, based on other evaluations, could include:

Changes in behaviour and attitudes of individuals, families and neighbourhoods (based on a range of indicators)

Capacity of FRC and community (based on a range of indicators)

Local and national service capacity (including inter-agency working)

‘Outcomes’ are the seven nationally are agreed objectives for children and families as summarised in the next slide

The big challenge in the evaluation is to refine this model and find robust ways of measuring each concept. There are also challenges for implementation.

Outcomes to be Achieved by FRC Programme Outcomes to be Achieved by FRC Programme Outcomes to be Achieved by FRC Programme Outcomes to be Achieved by FRC Programme

The FRC programme is aligned with national policy on children; in the absence of an explicit family policy, this is national family policy de facto.

There are seven national outcomes for children in the Agenda for Children’s Services (2007), building on the National Children’s Strategy (2000). Every child should be:

1. Healthy, both physically and mentally

2. Supported in active learning

3. Safe from accidental and intentional harm

4. Economically secure

5. Secure in the immediate and wider physical environment

6. Part of positive networks of family, friends, neighbours and community

7. Included and participating in society

The FRC framework has adopted these as outcomes for both children and their parents.

Measuring OutcomesMeasuring OutcomesMeasuring OutcomesMeasuring Outcomes

Outcome Possible Sources of Comparative DataPossible Sources of Comparative Data

1. Healthy GUI 2007-9; SLAN 2007; HBSC 2006GUI 2007-9; SLAN 2007; HBSC 2006

2. Actively learning GUI 2007-9; DES; NEWB 2004/5; GUI 2007-9; DES; NEWB 2004/5;

3. Safe from harm HSE or DOH; GUI 2007-9; HSE or DOH; GUI 2007-9;

4. Economic security SILC 2009; QNHSSILC 2009; QNHS

5. Safe environment GUI 2007-9GUI 2007-9

6. Positive networks GUI 2007-9; SLAN 2007; HBSC 2006GUI 2007-9; SLAN 2007; HBSC 2006

7. Participating in society GUI 2007-9GUI 2007-9

A major challenge for both the FRC strategy and its evaluation is that there is no agreed method for measuring national outcomes for purposes of monitoring and evaluation.

Some outcomes (eg 3, 4, 5, 7) do not seem to lend themselves to assessing the effectiveness of child and family services such as FRCs.

Next StepsNext StepsNext StepsNext Steps

Further work will involve:

Working with an advisory group of 4 FRCs to identify a set of ‘substantial and sustained interventions’ (2-4) that could be the subject of evaluation.

Developing the logic model for these interventions.

Finding instruments to measure the concepts, including availability of comparative data.

Developing a sampling framework and method of data analysis

Case studies could have a useful role within this framework as illustrations of pathways to outcomes that have already been established as effective by the evaluation.

These steps will lead to the preparation of a paper for the FSA detailing the framework for the evaluation.

ENDS HERE

GUI Instruments: Parent (9-yr old cohort)GUI Instruments: Parent (9-yr old cohort)GUI Instruments: Parent (9-yr old cohort)GUI Instruments: Parent (9-yr old cohort)

1. Socio-economic and demographic data on parents/caregivers, including social class, education, income, employment status, marital status, housing type, family type, support networks, friendship networks, leisure activities, childcare arrangements

2. DAS Relationship with partner by primary and secondary caregiver

3. Physical and mental health, including CES-Depression by primary and secondary caregiver

4. Neighbourhood problems by primary caregiver

5. Neighbourhood services by primary caregiver

Note that GUI is weak on the measurement of individual parent well-being despite the fact that it has substantial influence on child well-being

GUI Instruments: Child (9-yr old cohort)GUI Instruments: Child (9-yr old cohort)GUI Instruments: Child (9-yr old cohort)GUI Instruments: Child (9-yr old cohort)

1. Child’s EAS [Emotionality, Activity, and Shyness] Temperament by caregiver

2. Pianta child-parent relationship by primary and secondary caregiver

3. Child’s SDQ by teacher and caregiver

4. Child’s academic performance by teacher and primary caregiver

5. Drumcondra Reading & Maths competence by child

6. Attachment to school and teachers by child

7. Bullying and being bullied by child

8. Piers-Harris Self-Concept by child

9. Parenting Style Inventory II by child

10. Neighbourhood problems by child