Impact Evaluation of the

Opening Doors Community

Leadership Program for

Social Inclusion

FINAL REPORT

2

Dr Lucio Naccarella, PhD

Health Systems & Workforce Unit, Centre for Health Policy

Melbourne School of Population & Global Health

The University of Melbourne

August 2016

3

Acknowledgements

This evaluation was funded by Inner East Primary Care Partnership. We particularly thank the

Opening Doors Steering Committee and all the Graduates who participated in this evaluation as

without their cooperation this work would not have been possible.

For research correspondence please contact: Lucio Naccarella, PhD l.nacccarella@unimelb.edu.au

4

Contents Acknowledgements ................................................................................................................................................ 3

Executive Summary ................................................................................................................................................ 6

Introduction ....................................................................................................................................................... 6

Evaluation Focus ................................................................................................................................................ 6

Evaluation Methodology .................................................................................................................................... 6

Evaluation Findings ............................................................................................................................................ 8

1. Opening Doors Graduate Evaluation Participants ................................................................................ 8

2. Impact of Opening Doors ...................................................................................................................... 8

3. Benefits to Opening Doors Participants .............................................................................................. 10

4. Critical ingredients for success ............................................................................................................ 11

5. Reflections on Evaluation Methodology ............................................................................................. 11

Key Emerging Propositions .............................................................................................................................. 12

Opening Doors Program leadership capacity .............................................................................................. 12

Opening Doors Program Profile .................................................................................................................. 12

Social Inclusion Leadership capacity ........................................................................................................... 12

Opening Doors Monitoring and Evaluation Framework ............................................................................. 12

Introduction .......................................................................................................................................................... 14

Evaluation Methodology ...................................................................................................................................... 16

Opening Doors Program Logic Model .............................................................................................................. 18

Evaluation Findings ............................................................................................................................................... 19

Opening Doors Graduates Evaluation Participants .......................................................................................... 19

Impact of Opening Doors ................................................................................................................................. 22

Opening Doors Graduates perspectives...................................................................................................... 22

Opening Doors Steering Committee perspectives ...................................................................................... 30

Opening Doors Contextual factors ................................................................................................................... 30

Opening Doors Graduates perspectives...................................................................................................... 30

Opening Doors Steering Committee perspectives ...................................................................................... 33

Synthesis and Discussion of Evaluation Findings .................................................................................................. 34

Impact of Opening Doors ................................................................................................................................. 34

Impact in the Community ........................................................................................................................... 34

Impact on Reducing Social Isolation ........................................................................................................... 37

Benefits to Opening Doors Participants ........................................................................................................... 39

Critical ingredients for success ........................................................................................................................ 41

Reflections on Evaluation Methodology .......................................................................................................... 41

Ripple Effect Mapping ................................................................................................................................. 41

5

Opening Doors Program Logic Model ......................................................................................................... 42

Key Emerging Propositions ................................................................................................................................... 43

Opening Doors Program leadership capacity .................................................................................................. 43

Opening Doors Program Profile ....................................................................................................................... 43

Social Inclusion Leadership capacity ................................................................................................................ 43

Opening Doors Monitoring and Evaluation Framework .................................................................................. 44

Appendix A: Ripple Effect Mapping Workshop .................................................................................................... 45

Appendix B: Opening Doors Program Logic Model .............................................................................................. 46

Appendix C: Ripple Effect Mapping Reporting Template - Examples Only ........................................................... 47

Appendix D: Community Capitals Framework ...................................................................................................... 48

Appendix E: Initial Draft Ripple Effect Map (May, 2015) ...................................................................................... 50

Appendix F: Reflections on Creating a Culture of Social inclusion ....................................................................... 51

6

Executive Summary Introduction The Opening Doors Community Leadership Program for social inclusion (Opening Doors) was

established in 2009 in the Eastern Metropolitan Region of Melbourne, Victoria, to build leaders in

local communities with the knowledge, skills, resources and networks to create more socially

inclusive communities. Opening Doors brings together a diverse range of community members of

different ages, faiths and cultural and social backgrounds to develop their leadership skills,

knowledge and insights about social isolation. A key component of Opening Doors is support for

each participant, with their newfound knowledge and skills, to work within their community to

establish a local community-based project to strengthen community cohesion and connectedness.

Since 2009 Opening Doors has supported 120 graduates to successfully complete the program and

generate multiple community-based projects in the Eastern Metropolitan Region of Melbourne.

Evaluation Focus In 2015 the auspice and Steering Committee of Opening Doors funded the University of Melbourne

to assess the impacts and benefits of Opening Doors on its graduates and beyond, and to identify the

critical ingredients for the success of Opening Doors. More specifically the proposed research

question was:

How successful has the Opening Doors leadership program

been in reducing social isolation?

The key objectives of the evaluation were:

To capture and describe the long term impacts of the program in reducing social isolation

To capture the ways in which the program has been of benefit to the participants

To identify the critical ingredients for success of the program.

Evaluation Methodology Evaluating changes in individuals, groups, organisations, or communities resulting from community

development programs are recognised as being challenging. Traditional impact evaluation,

experimental, and other quantitative methods are difficult to implement and often provide limited

insight into the process that brought about change, or the context of change observed as a result of

the community development program. Participatory qualitative impact evaluation methods are

considered better suited to evaluate community development programs.

Given that Opening Doors is a community development leadership program (empowering

community leaders, recognising and valuing diversity, and building on individual and local

community strengths), to address the research question and evaluation objectives an emerging

participatory qualitative impact evaluation approach was chosen: Ripple Effect Mapping. Ripple

Effect Mapping uses elements of appreciative inquiry, mind mapping, and qualitative open-ended

group interviewing to engage program participants and other community stakeholders to reflect

upon and visually map the intended and unintended changes produced by programs. Ripple Effect

Mapping has been used to evaluate multiple community-based development programs designed to

strengthen leadership, to reduce poverty, and to build social capital amongst youth.

7

Ripple Effect Mapping is traditionally conducted in five key steps:

1. Schedule a Ripple Effect Mapping workshop;

2. In workshop conduct interviews in pairs;

3. In workshop brainstorm and map using software the ripples from a program;

4. After workshop conduct follow-up interviews;

5. Clarify, code, and analyse data.

Due to resource and time constraints and to address the research question and evaluation

objectives, an adapted Ripple Effect Mapping process was used, with three key steps:

Step 1: Schedule Ripple Effect Mapping workshop with:

Opening Doors Steering Committee members: From Link Health and Community; Inner East

Primary Care Partnership; Carrington Health; and Deakin University. This workshop served

to provide contextual information for the evaluation.

Opening Doors Graduates: Given that since 2009 there have been 120 Opening Doors

graduates or approx. 20 participants annually, to capture the diversity of graduate

experiences (not necessarily representativeness) a two-tier sampling framework was used.

Potential Opening Doors graduate evaluation participants were initially clustered by Opening

Doors Coordinator into three groups based on graduating year: 1) 2009 - 2011; 2) 2012-

2013; 3) 2014 – 2015. Participants were then selected based on: their personal features

(age, faith, cultural background); whether they were linked to an organisation; and the type

of community project developed.

Based upon these criteria 58 Opening Doors graduates were identified and invited via email by the Opening Doors Coordinator to participate in one Ripple Effect Mapping workshop, lasting approximately 90 minutes. Graduates were provided with 12 possible workshop sessions to choose from to optimise attendance. Upon agreement, the evaluator emailed the graduates - thanking them for agreeing to participate, confirming workshop date and time, and including a plain language statement, a consent form and a workshop outline.

Step 2: Conducted Ripple Effect Mapping Workshops with the Opening Doors Steering Committee

and then Opening Doors graduates to brainstorm and map the ripple effects of ‘Opening Doors’. To

set the scene, all workshops participants were initially asked to reflect upon a statement: ‘Creating a

Culture of Social inclusion’ using a framework: values, symbols, practices and systems. This activity

was then followed using a workshop outline (Appendix A) that posed the following five questions:

1. What drivers for taking action on social inclusion exist?

2. What actions have Opening Doors graduates taken to create more socially inclusive

communities?

3. What changes have Opening Doors graduates noticed specifically related to social inclusion?

4. What enablers and barriers exist to the Opening Doors program?

5. What is required to sustain the changes in socially inclusive communities?

Step 3: Transcribe and analyse data: After all the Ripple Effect Mapping workshops, audio-recorded

discussions were sent to an independent transcribing service to be transcribed. The transcripts were

then analysed using the constant comparative thematic analysis approach.

To assist in clarifying how Opening Doors worked and to refine the funded evaluation plan, questions

and data collection methods, a program logic model (Appendix B) was also developed in February

2016.

8

The evaluation received ethics approval from The University of Melbourne Human Ethics Advisory

Group.

Evaluation Findings

A summary of the evaluation findings are presented under five headings:

1. Opening Doors graduate evaluation participants

2. Impact of Opening Doors

3. Benefits to Opening Doors Participants

4. Critical ingredients for success

5. Reflections on evaluation methodology.

1. Opening Doors Graduate Evaluation Participants A total of 58 Opening Doors graduates were identified and invited by the Opening Doors Coordinator

to participate in one Ripple Effect Mapping workshop. Of these 26 (49%) participated in the

workshops, or 22% of the 120 total graduates across the last seven years. Workshop participants

were: predominantly female (n=17), reflecting Opening Doors participant profiles, from across seven

year levels (2009 to 2015) - although more of the recent graduates agreed to participate. Graduates

were also from a range of ages, faiths, cultural backgrounds, and predominantly not employed or

formally linked to an organisation. Graduates had a broad range of drivers/motivations for taking

action on social inclusion and participating in Opening Doors that can be summarised into: Self-

oriented drivers (personal enhancement; career opportunities) and Others-oriented drivers (acting

on humanitarian values; building understanding about others; developing social networks).

2. Impact of Opening Doors Impact in the Community

The Ripple Effect Mapping workshops with the 26 Opening Doors graduates revealed that since 2009

Opening Doors has successfully recruited and supported a diverse group of local community leaders

to transform participant’s ideas into projects to create more socially inclusive communities.

Opening Doors graduates have taken multiple actions to create more socially inclusive communities

that can be summarised into three levels:

Individual level: established projects; championed access issues; established connections,

accessed facilities; advocated for Opening Doors; applied for and obtained employment

positions;

Organisational level: established structures, committees and partnerships; contributed to

strategic planning processes; took up advisory roles on disability organisations; took up

leadership roles in senior citizen organisations; supported strategic plans; contributed to

disability access policies; and

Community level: applied for funding to conduct projects; established and facilitated

support groups; developed resources (books) for communities; connected community

groups.

Opening Doors has benefitted multiple individuals and communities, including: older people; Indian

senior citizens; people with visual impairment; people with chronic diseases such as diabetes;

parents in general; parents of children with Autism Spectrum Disorder (ASD); people with anxiety,

depression and mental illness; newly-arrived migrants; refugee children; transgender people;

parents of transgender families; people with gambling addictions; ex-offenders; people with a

disability; and carers of people with health conditions or other social disadvantage.

9

The benefits for communities can be clustered into five categories, including:

1. Access - Increased access to technology;

2. Knowledge - increased knowledge of health conditions, local services, support systems for

carers;

3. Capabilities - increased self-esteem, increased confidence;

4. Advocacy - increased spaces to be heard and have a voice for; carers, parents of children

with ASD, people with anxiety and depression, and parents of transgender families;

5. Connections - increased connections between newly arrived migrants and local Australians.

Opening Doors has also benefitted health professionals (e.g. trainee doctors); health service

organisations (e.g. RDNS); cultural specific organisations (e.g. Indian Senior Citizens Association);

Residential Aged Care organisations (e.g. DonCare); non-government organisations (e.g. St Vincent’s

De Paul, Jesuit Social Services); and local government organisations (e.g. Maroondah City Council).

The benefits for those professionals and organisations can be clustered into two main categories:

1. Knowledge: increased knowledge of issues facing: carers; parents of children with ASD; ex-

offenders; people with a mental illness, people with disabilities, and parents of transgender

families;

2. Connections & Collaborations: increased connections and collaborations between Senior

Citizen Associations; non-government organisations to support youth leadership.

Opening Doors is contributing to changes in the broader community that can be clustered into eight

domains:

1. Increasing the awareness and knowledge of factors that contribute to social isolation, social

exclusion and those factors that promote social inclusion;

2. Increasing the quality and number of relationships, connections, networks and

collaborations;

3. Increasing ideas, viewpoints, mindsets, and ways of thinking about social inclusion using

asset-based community development approaches;

4. Increasing the leadership capabilities; increased empowerment, confidence and advocacy;

5. Increasing the level of participation and engagement of marginalized local communities;

6. Increasing the level of respect and acceptance of people who are marginalised and socially

excluded from society;

7. Increasing the level of belonging and decreasing the level of loneliness of marginalized

communities;

8. Increasing the level of volunteering, generosity, goodwill and benevolence - giving back to

others.

Overall these evaluation findings confirm previous evaluations of Opening Doors by Held (2011)

which identified two key outcomes of Opening Doors, namely: increased participant’s leadership

knowledge, skills and confidence to engage with communities; and establishment of community-

based projects to strengthen community connectedness and reduce social isolation in local

neighbourhoods across the inner east catchment of Melbourne.1

1 Held, R. (2011). Opening Doors: A community leadership program for social inclusion. Evaluation Report. Inner East Primary care Partnership. http://iepcp.org.au/sites/default/files/Report%20IESII%20Evaluation%20Final_0.pdf

10

The Ripple Effect Mapping evaluation workshops have further revealed that Opening Doors is

contributing to building community capacity as demonstrated by: the development local leadership;

improving community participation; improving local assessment of problems and assets; improving

local resource mobilization; strengthening local community connections, collaborations and

partnerships; building local structures; and building local project capabilities.

The findings from the Ripple Effect Mapping workshops and the design of Opening Doors also

resonates with the work on Community Capitals (i.e. all of the things in a community that have the

potential to be a resource that can be invested, saved, or used up).

Impact on Reducing Social Isolation

Based on the collective impact of the Opening Doors graduates projects, actions and ripples in the

community, one could surmise that Opening Doors is contributing to reducing social isolation.

However, caution is required in making a ‘cause and effect’ judgement. As a community

development and empowerment program, Opening Doors is designed to influence multiple social

phenomena (social exclusion, social inclusion, social isolation, social cohesion, social capital,

community participation, marginalisation, and community engagement) – all of which have

conceptual, definitional and measurement issues. Multiple quantitative indicators of social isolation

exist, as do multiple scales and indexes that measure issues that intersect with social isolation and

social exclusion.

To provide a definitive assessment of whether Opening Doors is reducing social isolation, a

quantitative evaluation approach using validated instruments would have been required.

Drawing upon the Ripple Effect Mapping workshops, we can speculate that Opening Doors is

contributing to addressing proxy indicators of social isolation as suggested by Zaveleta et al (2014),

namely: external social inclusion (e.g. increased the frequency of social contact; increased social

network support; increased reciprocity; and volunteering) and internal social isolation (e.g. increased

satisfaction with social relations; increased need for relatedness; increased feeling of belonging;

decreased loneliness; and increased trust).

3. Benefits to Opening Doors Participants The Ripple Effect Mapping workshops have confirmed previous evaluations of Opening Doors that it

is building its participants’ leadership capability as demonstrated by:

Supporting and building its participants’ leadership assets and strengths to inspire others to

work with them to create a greater sense of community that is inclusive of age, ability and

cultural diversity;

Strengthening its participants’ relationships, connections and networks to transform ideas

and to design, plan and implement projects to increase social inclusion and address social

inclusion issues in their local community;

Supporting its participants’ personal leadership knowledge, skills and resources to use and

apply asset/strength-based community development approaches to address the social

isolation that people experience.

The workshops have also revealed that Opening Doors is developing both the skills and capacity of

individuals (leader or human capital development) and the collective capacity of groups and the

community (leadership or social capital development).

11

4. Critical ingredients for success The Ripple Effect Mapping workshops with both the graduates and Steering Committee members

revealed the following key ingredients for the success of Opening Doors:

A Coordinator:

Who has core relational qualities (benevolent, non-judgmental, honest, open, goodwill,

reliable, accepting, and personable);

Who has connections to multiple organisations and networks and know-how;

Who has lived experience in relation to having been an Opening Doors participant;

Who has experience putting into practice the Asset-Based Community Development (ABCD)

approach to community development.

A Program

that is free and based upon the principle of volunteerism, and open to local community

members with a combination of self- and other-oriented drivers for taking action on social

inclusion;

that has a structure, including an initial live-in retreat (providing a safe environment for

building relationships), followed by interactive workshops with activities and dedicated

spaces and time for reflection; and ending with a graduation that showcases graduates

actions and impact of their community-based projects;

that has content that is pragmatic and practical and includes core resources for building

skills and capabilities in: leadership; asset-based community development (ABCD); team

building; project development; events and media management. Opening Doors graduates

overwhelmingly valued and commented that the Asset-Based Community Development

(ABCD) approach to community development was a key enabler to the success of Opening

Doors, identifying, developing and mobilising existing strengths (assets) in the community,

building community networks, and transforming ideas, mindsets, assumptions and

expectations into community-based projects to address local needs;

that is auspiced and supported by a Steering Committee comprised of a network of

organisations;

that is not project-based but designed to build community leaders and leadership;

that is supported and authorised by a wide spectrum of health, human and social service

organisations.

A Post-Program Strategy

That supports its graduates with social and skill-based opportunities (virtual, face to face,

forums) to maintain and establish new connections, share new knowledge and ideas, and

to reflect with other graduates

5. Reflections on Evaluation Methodology The Ripple Effect Mapping workshops generated rich evidence about the complexity of Opening

Doors and the actions and effects resulting from its graduates. As a participatory evaluation method,

the Ripple Effect Mapping workshops accentuated the graduates relationships and connections

developed while participating in Opening Doors. The method assisted graduates to express their

voice and illustrate how their ideas were transformed into community-based projects and

contributed to creating more socially inclusive communities. The Ripple Effect Mapping workshops

also provided graduates with further opportunities to connect with other past graduates.

12

More specifically the workshops contributed to building social capital among graduates - bonding

capital (re-connecting with known graduates) and bridging capital (connecting with new graduates)

to put ideas into action. The information from the workshops can inform future evaluations and

questions to assist the work of Opening Doors.

While the Ripple Effect Mapping methodology has multiple benefits, we acknowledge that a

limitation is the risk of bias in participant selection, resulting in selective or incomplete information

about outcomes or consequences – both positive and negative. Given the rich, frank and honest

views expressed by all graduates in the workshops about Opening Doors, we consider the

methodology as successful in capturing the diversity of graduate experiences.

Key Emerging Propositions To optimise the investment and sustainability of Opening Doors and its ripple effects, the following

propositions could be considered:

Opening Doors Program leadership capacity 1. A shared Opening Doors Coordinator model could be considered to enable a shared

workload, to build shared trust and a supportive platform for past and current Opening Doors Graduates social inclusion leadership;

2. An Opening Doors alumni mentoring model could be considered to formally support the Coordinator to support current participants and to reduce the current dependency upon the Coordinator for ongoing support;

3. The role of the Opening Doors Steering group as a formal structure to support the Opening Doors Coordinator and the Programs ripple effects could be further explored;

4. The role and contribution of Link Health and Community Service as a formal structure to support the Opening Doors Coordinator could be further explored.

Opening Doors Program Profile 5. Accrediting and aligning Opening Doors with other Leadership programs (e.g. Global

Leadership Foundation) could be explored as a way of further providing an explicit public profile for Opening Doors;

6. Given that the local community-focused approach of Opening Doors was valued by all graduates, promoting the local ripple effects to a broader health, social and community service base could be considered.

Social Inclusion Leadership capacity 7. To sustain the momentum of Opening Doors graduates establishing an Opening Doors

Alumni Community of Interest, facilitated by alumni, to provide social connection and skills-based opportunities (virtual, face to face, forums) could be explored;

8. Investing in an expanded Opening Doors social media platform (webpage, facebook etc) to share ideas and action of all past and present Opening Doors participant could be considered;

9. Given that Opening Doors is a free program, and reliant on volunteers, sustainable strategies to optimise the recruitment and retention of its volunteer community members need to be considered.

Opening Doors Monitoring and Evaluation Framework 10. Given that Opening Doors is building community capacity, future evaluations of Opening

Doors could utilise recognized community capacity building dimensions as an ongoing monitoring and evaluation framework;

11. Given that Opening Doors is investing in resources and assets (Capital) the Community Capitals framework could be considered as an ongoing planning, monitoring and evaluation framework;

13

12. Given that the Ripple Effect Mapping methodology contributes to engaging graduates and illuminating effects, evaluating project specific changes using Ripple Effect Mapping could be considered;

13. To assess definitively whether Opening Doors is reducing social isolation, future evaluations of Opening Doors could consider utilising Zavelata’s et al (2014) social isolation conceptual framework and validated instruments.2

2 Zavaleta, D., Samuel, K. & Mills, C. (2014). Social isolation: A conceptual and measurement proposal. OPHI Working Papers 67. http://www.ophi.org.uk/social-isolation-a-conceptual-and-measurement-proposal/

14

Introduction The Opening Doors Community Leadership Program for social inclusion (‘Opening Doors’) was

established in 2009 in the Eastern Metropolitan Region of Melbourne, Victoria, to build leaders in

local communities with the knowledge, skills, resources and networks to create more socially

inclusive communities. Opening Doors is modeled on the Leadership Victoria ‘Williamson

Community Leadership’ model based on the notion that investing in a diverse group of community

leaders will then give back through their contribution to local projects.3 Opening Doors is premised

upon the following assumptions:4

Investing in community leaders was likely to have a greater impact than funding individual community projects;

By building leadership capacity, the program would continue to benefit the community over many years through the contribution of these leaders;

Potentially isolated people can be hard to reach and grass roots community leaders are more likely to be able to identify and engage those people in their local communities;

Initial research showed that while agencies were concerned about the impacts of social isolation, their response was often to refer people to other services, rather than linking with local community groups;

Communities often have the capacity to provide solutions from within their own community, utilising the available assets (both human and physical) without necessarily requiring funding and services;

By bringing together a diverse group, participants will gain from each other and increase their contacts and networks.

Opening Doors is underpinned by asset-based community development approaches to support

community members to become change agents.5 6 Opening Doors brings together a diverse range of

community members of different ages, faiths and cultural and social backgrounds to develop their

leadership skills, knowledge and insights about social isolation. While the structure and content of

Opening Doors is iterative and responsive to the particularly years graduates, it has usually included

an initial two to three day live-in retreat, followed by workshops over a six month period on topics

including: leadership; asset-based community development; social isolation, team building; project

development; events and media management; and ending with a graduation that showcases a

sample of graduates community-based projects. A key component of Opening Doors is supporting

each participant with their newfound knowledge and skills to work within their community to

establish local community-based projects to strengthen community cohesion and connectedness.

Since 2009 Opening Doors has supported 120 graduates to successfully complete the program and

generate multiple community-based projects in the Eastern Metropolitan Region of Melbourne.

3 Leadership Victoria Williamson Community Leadership Program. www.leadershipvictoria.org.au 4 Held, R. (2011). Opening Doors: A community leadership program for social inclusion. Evaluation Report. Inner East

Primary care Partnership. http://iepcp.org.au/sites/default/files/Report%20IESII%20Evaluation%20Final_0.pdf 5 Kretzmann, J. P., & McKnight, J. L. (1993). Building communities from the inside out: a path toward finding and mobilizing

a community’s assets. Evanston, Ill.: Center for Urban Affairs and Policy Research, Northwestern University.

http://www.abcdinstitute.org/publications/basicmanual/. 6 Kretzmann, J. P. (2010). Asset-based strategies for building resilient communities. In J. W. Reich, A. Zautra & J. S. Hall (Eds.), Handbook of adult resilience. New York: Guilford Press

15

In 2015 the auspice and Steering Committee of Opening Doors funded the University of Melbourne

to assess the impacts and benefits of Opening Doors to its graduates and beyond, and to identify the

critical ingredients for the success of Opening Doors. More specifically, the proposed research

question was:

How successful has the Opening Doors leadership program

been in reducing social isolation?

The key objectives of the evaluation included:

To capture and describe the long term impacts of the program in reducing social isolation;

To capture the ways in which the program has been of benefit to the participants;

To identify the critical ingredients for success of the program;

To ensure that the evaluation methodology and findings are contextualised this next section

presents a brief overview of impact evaluation approaches appropriate for evaluating community

development program such as Opening Doors.

Evaluating changes in individuals, groups, organisations, or communities resulting from community

development programs are recognised as being challenging, yet demonstrating impacts is critical for

continued investment.7 Community development programs traditionally seek to empower the

community, recognising and valuing diversity, and build upon the individual and local community

strengths. Adhering to the interdependence principle of systems theory and ecological theory,

planned multilevel community interventions often create unanticipated ripples in the community;

which either go unobserved, unappreciated, or simply undocumented.8 Traditional impact

evaluation, experimental, and other quantitative methods are difficult to implement in efforts to

document the effectiveness and outcomes of community-based development programs.

Furthermore, experimental and other quantitative methods often provide limited insights into the

process that brought about change or the context of change observed as a result of the community

development programs.

Other approaches to impact evaluation exist including:

Objectives-based evaluation: Investigating whether the stated objectives of the program have been achieved;

Needs-based evaluation: Identifying the needs which the program responded to and investigate to what extent those needs have been met;

Goal-free evaluation: Taking an arms-length ‘open’ approach, considering any effects;

Theory-based evaluation: Using program logic to establish expected outcomes and investigate to what extent those outcomes have been achieved;

Participatory evaluation: Participatory approaches are intended to empower beneficiaries by enabling them to shape decisions which affect their lives;

Counterfactual impact evaluation: Comparing the difference between what happens in the presence of the program with what occurs in the absence of the program.

There is no one right way to conduct an impact evaluation. What is needed is a combination of

methods and designs that suits the particular situation. When choosing these methods and designs,

7 Rennekamp, R. A., & Arnold, M. E. (2009). What progress, program evaluation? Reflections on a quarter-century of Extension evaluation

practice. Journal of Extension, 47(3): http://www.joe.org/joe/2009june/comm1.php 8 Trickett, EJ (2009). Multilevel Community-Based Culturally Situated Interventions and Community Impact: An Ecological Perspective.

American Journal of Psychology. 43:257-266.

16

three issues need to be taken into account: the available resources and constraints; the nature of

what is being evaluated; and the intended use of the evaluation. Appropriate impact evaluation

design requires situational responsiveness – matching the design to the needs, constraints, and

opportunities of the particular case.9 Patton (2006) has argued that it is important to maximize

community engagement by involving communities in evaluation methods that inform and motivate

them.10 Ripple Effect Mapping has recently emerged as a participatory qualitative impact evaluation

method suited for the evaluation of community development programs.11

Evaluation Methodology Given that Opening Doors is a community development leadership program, empowering community

leaders, recognising and valuing diversity, and building on individual and local community strengths,

to address the research question and evaluation objectives, Ripple Effect Mapping was chosen as

the participatory qualitative impact evaluation approach.

Ripple Effect Mapping (REM) uses elements of appreciative inquiry, mind mapping, and qualitative

open-ended group interviewing to engage program participants and other community stakeholders

to reflect upon and visually map the intended and unintended changes produced by programs.12

REM is not only a powerful technique to document impacts, but a way to engage and re-energize

program participants, increasing the likelihood of future collective action.13

Three approaches to Ripple Effect Mapping exist:

1. ‘Web mapping’ where the group session examines short-term, medium-term, and long-term

impacts and maps them directly onto a mind map;

2. ‘In-depth rippling’ where the group session focuses on the deepest and most impactful

chains of events; and

3. ‘Theming and rippling’ where the group session captures the breadth of reporting impacts

from all participants, generates impact themes, and examines ripples once themes are

generated.

Ripple Effect Mapping is traditionally conducted in five key steps:

1. Schedule a Ripple Effect Mapping Workshop involving participants from the program. A

group of 8-15 is recommended.

2. In the Ripple Effect Mapping workshop conduct interviews using Appreciative Inquiry

questions to start the conversation. Appreciative Inquiry is a group facilitation method that

invites people to reflect on the most positive aspects of a situation. At the start of a ripple

effect mapping session, participants pair up and interview each other about ways they or

their organisation were positively affected by the leadership course. The interviews serve as

an ice-breaker to prepare participants for the group mapping session.

9 Rogers, PJ (2009). Matching impact evaluation design to the nature of the intervention and the purpose of the evaluation. Journal of

Development Effectiveness. 1(3): 217–226 10 Patton, M.Q. (2006) Evaluation for the Way We Work.The Nonprofit Quarterly. Vol. 13 (1): 28-33. 11 Kollock, DA (2011). Ripple effects mapping for evaluation. Washington State University Curriculum. Pullman, WA 12 Kollock, DA (2011). Ripple effects mapping for evaluation. Washington State University Curriculum. Pullman, WA. 13 Baker, B., Calvert, M., & Emery, M. (2011). Mapping your impact in the community. National 4-H Council Engaging Youth Serving

Community: Tools for Evaluation of Your EYSC Project. Retrieved from http://fyi.uwex.edu /youthadultpartnership/2012/05/30/tool-for-

mapping-community-impac Nathaniel, K., & Kinsey, S. (2013). Contributions of youth engagement to the development of social capital

through community mapping. Journal of Extension, 51, 1.

17

3. In the Ripple Effect Mapping workshop hold a group mapping session. The core of a Ripple

Effect Mapping session involves group mapping – a process of brainstorming and recording

the effects (the ‘ripples’) of an initiative – either through mind mapping software or notes

taped to a wall. This process engages the entire group and enables participants to see the

connections among project or program effects (as well as to continue building personal

relationships). Ideally a facilitator and a mapper co-lead the mapping session, which lasts

from one to two hours. The resulting ‘map’ visually depicts the ripple effects of the initiative.

4. After workshop conduct follow-up interviews with course participants, executive sponsors

or key organisation members to supplement ripple effect map (e.g. enablers and barriers,

what else require to sustain changes etc).

5. Clarify, code, and analyse data. After the session, the evaluator codes and analyses data

into short-term, medium-term or longer-term changes drawing upon the visual ripple effects

map.

Ripple Effect Mapping has been used to evaluate multiple community development programs. For

example, Ripple Effect Mapping was used to conduct an impact evaluation of the Horizons program,

an 18-month community-based program delivered to strengthen leadership to reduce poverty.14

Ripple Effect Mapping has also been used to evaluate the impacts of youth programs on building

social capital.15 Ripple Effect Mapping enabled the youth program participants to describe the

connections they've built, as well as what these connections led to. The limitations of Ripple Effect

Mapping are the risk of bias in participant selection and in data collection. The assembled

participants may also not have complete information about all the outcomes of a program and may

not provide examples of negative consequences.

Due to resource and time constraints an adapted Ripple Effect Mapping process was used, with

three key steps:

Step 1: Schedule Ripple Effect Mapping workshop with:

Opening Doors Steering Committee members from: Link Health and Community; Inner East Primary Care Partnership; Carrington Health; and Deakin University. This workshop served to provide contextual information for the evaluation.

Opening Doors Graduates. Given that since 2009 there have been 120 Opening Doors graduates, approximately 20 participants annually, to capture the diversity in graduate experiences (not necessarily representativeness) a two-tier sampling framework was used. Potential Opening Doors graduate evaluation participants were initially clustered by Opening Doors Coordinator into three groups upon graduation year: 1) 2009 - 2011; 2) 2012 - 2013; 3) 2014 - 2015, and then selected based on their; personal features (age, faith, cultural background), whether they were linked to an organisation, and the type of community project developed. Based upon these criteria 58 Opening Doors graduates were identified and invited via email by the Opening Doors Coordinator to participate in one Ripple Effect Mapping workshop, lasting approximately 90 minutes. Graduates were provided with 12 possible workshop sessions to choose from to optimise attendance. Upon agreement, the evaluator emailed the graduates - thanking them for agreeing to participate, confirming

14 Welborn et al (2016) Turning the Tide on Poverty: Documenting impacts through Ripple Effect Mapping, Community Development, 47:3,

385-402, DOI:10.1080/15575330.2016.1167099 15 Baker, B and Johannes (2013). Measuring social capital using ripple mapping. New Directions for Youth Development. 138: 31-47;

Welborn et al (2016). Turning the tide on poverty: Documenting impacts through ripple effect mapping. Community Development. 47:3,

385-402

18

workshop date and time, and including a plain language statement, a consent form and a workshop outline.

Step 2: Conducted Ripple Effect Mapping Workshops with the Opening Doors Steering Committee

and then Opening Doors graduates to brainstorm and map the ripple effects of Opening Doors. To

set the scene, all workshops participants were initially asked to reflect upon a statement: ‘Creating a

Culture of Social inclusion’ using a framework: values, symbols, practices and systems. This activity

was then followed using a workshop outline (Appendix A) that posed the following five questions:

1. What drivers for taking action on social inclusion exist?

2. What actions have Opening Doors graduates taken to create more socially inclusive

communities?

3. What changes have Opening Doors graduates noticed specifically related to social inclusion?

4. What enablers and barriers exist to the Opening Doors program?

5. What is required to sustain the changes in socially inclusive communities?

Step 3: Transcribe and analyse data: After all the Ripple Effect Mapping workshops, audio-recorded

discussions were sent to an independent transcribing service to be transcribed. The transcripts were

then analysed using the constant comparative thematic analysis approach.16

The evaluation received ethics approval from The University of Melbourne Human Ethics Advisory

Group.

Opening Doors Program Logic Model To assist in clarifying how Opening Doors worked and refining the funded evaluation plan, questions

and data collection methods, a program logic model (Appendix B) was developed in February 2016.

Program logic models provide visual representation of the assumptions about how a program is

supposed to work, and the causal linkages between context, inputs, activities, outputs, and

outcomes .17

16 Strauss, A. & Corbin, J. (1998). Basics of Qualitative Research. Techniques and Procedures for Developing Grounded Theory. London 17 Funnell S. (1997). Program logic: An adaptable tool for designing and evaluating programs, Evaluation News and Comment; 6 (1)

19

Evaluation Findings This section presents the evaluation findings, organised into three sections:

1. Opening Doors graduate evaluation participants

2. Impact of Opening Doors in the community

3. Opening Doors contextual factors (enablers and barriers).

Opening Doors Graduates Evaluation Participants A total of 58 Opening Doors graduates were identified and invited by the Opening Doors Coordinator

to participate in the Ripple Effect Mapping workshops. Of these 26 (49%) actually participated in the

workshops, or 22% of the 120 total graduates across the last six years. As can be seen from Table 1,

the individual workshops were small in size and had a diverse composition in terms of year

graduated and gender.

Graduate evaluation participants were: predominantly female (n=17), reflecting Opening Doors

participant profiles; from across all seven year levels (2009, 2010, 2011, 2012, 2013, 2014, 2015) -

although more of the recent graduates agreed to participate in the workshops. While this is not

shown in Table 1, graduates were also from a range of ages, faiths, cultural backgrounds, and were

predominantly not employed or formally linked to an organisation.

(Please note: the size and composition of the Ripple Effect Mapping workshops, were beyond the

sphere of control of the evaluator and reflects the availability of graduates to participate in the

scheduled workshops

20

Table 1: Opening Doors Graduate Evaluation Participant Profile

Workshop Group Participants

(n=26) Year and Gender

Female (F)

(n=17)

Male (M)

(n=9)

Key drivers for taking action on social inclusion exist and participating in Opening Doors?

Illustrative Quotes

Group 1: N=4

2009 - M

2011 - F

2013 - M

2015 – M

1 3 Wanted to act on human rights issues

Wanted to learn

We can act on social isolation

Increasing accessibility is an issue

Want to learn about community developments approaches

…I stopped thinking about myself. I thought what I can do for others? And that's where all my projects started coming out. Male Graduate 2013 My only reason for doing [Opening Doors] was that I was angry when I read the leaflet, that people weren't included. They were excluded. That people were left out. Female Graduate 2014 My motivation was that I had personally experienced depression and I have found a mentor who helped me with that. Female Graduate 2015 I felt a strong desire to support her in her work. Then Opening Doors was an extension of wanting to do that. I've definitely been able to support her and the whole organisation and myself. Female Graduate 2015

Group 2: N=3

2014 - F

2015 x 2 F, F

3 - Having personal depression

Having a passion for people and food

Wanted to connect

Wanted to volunteer

Being angry and want to act

Group 3: N=3

2012 x 2 - F, F

2015 – F

3 - Build my network

Wanted to learn what is in my community

Wanted to provide opportunities for others

Want to do community-based projects

Group 4: N=3

2014 - M

2015 x 2 - F, F

2 1 Commitment to seniors and young people

Personal family issues – disability

Already volunteers

Wanted to give back

Group 5: N=6

2011 - F

2012 - M

2013 x 3 - M,M,M

2015 – F

2 4 To create happiness amongst people, to join them together and fill the gap in their life

Personal and life experiences

Wanted to network

Wanted to give back

Group 6: N=4

2010 - F

2015x3 - F,F,M

3 1 Wanted to learn the ABCD community development approach

Want to use psychological skills more

Wanted to increase awareness about social inclusion

Wanted to move ideas into projects

Wanted to address stigmas

Group 7: N=3

2011 - F

2012 - F

2013 - F

3 - Personal experiences

Wanted to improve my leadership skills

Wanted to make ideas happen

Wanted to give back

Wanted to learn the ABCD approach

Wanted to make a difference and to socialise

21

Graduates also had a broad range of motivations for taking action on social inclusion and

participating in Opening Doors. As can be seen in Table 1, graduates had in-depth personal life

stories and journeys that led them to become receptive and ready for Opening Doors. Key drivers for

taking action can be summarised into self- or other-oriented drivers, within which there are five

domains.

Self-oriented drivers

1. Enhancement – seeking to feel needed and good about themselves (giving back; want to

lead a valued life; have anger but want to act; want to do);

2. Career – seek to obtain benefits to assist them with employment opportunities (with new

networks new employment opportunities exist).

Others oriented drivers

3. Values – seeking to express humanitarian values through actions (act on rights issues –

human, disability, abuse; want to increase access; recognise diversity of community – young,

old);

4. Understanding – seeking to learn more about the world, other people and develop their

own skills (desire to learn what is in the community; family situation- social role valorisation

– disability; want to learn; learn about new roles);

5. Social – seeking to build or reinforce bonds with others (social isolation – empathy; develop

local community-based networks).

The Ripple Effect Mapping workshop with the Opening Doors Steering Committee revealed a similar

array of self- and other-oriented drivers/motivations for taking action on social inclusion and

participating in Opening Doors, including:

Self-oriented: (for personal growth; to build and enable capability building to transform

ideas into reality)

Others oriented: (to innovate in a flexible supportive environment; to match ideas with

resources; to belong; to do justice; to support those with ideas, money and resources; to

put social inclusion in action; to put strength-based approaches supporting change into

practice).

22

Impact of Opening Doors This section provides findings from the Ripple Effect Mapping workshops, firstly from Opening Doors

graduates and then from the Opening Doors Steering Committee.

Opening Doors Graduates perspectives The seven Ripple Effect Mapping workshops with the 26 Opening Doors graduates generated

extensive insights into the multiple impacts of participating in Opening Doors. Table 3 - Table 9

provides a summary of the self-reported impacts within each workshop at three levels:

Ripple 1: (Specific actions taken by Graduates);

Ripple 2 (Who benefitted and how?);

Ripple 3 (How has the community changed?).

This template (i.e. table structure) for reporting Ripple Effect Mapping is based upon previous

evaluations of community development programs that have used the Ripple Effect Mapping

methodology.18 Appendix C provides an example of a Ripple Effect Mapping report table.

Please note: given that only 26 graduates (22% of all graduates) participated in the REM workshops, the actions and ripples are not an exhaustive list but illustrative of the spectrum of ripples or impacts

and value of Opening Doors.

Please note: when graduates were asked about specific actions they had taken, most graduates mentioned their specific project (e.g. Parent support Groups; Welcome Dinners project; Black dog project; GROWSAIL, U3A, Transfamily, Teen Dinners, WALKs; Monash Men's Shed etc). However,

several graduates commented that even though their ‘idea’ had not developed into a ‘Project’ worthwhile actions still occurred.

Please note: Ripple Effect Mapping workshops were not project specific, but designed to capture the diversity of graduate experiences and effects of participating in Opening Doors.

18 Welborn et al (2016) Turning the Tide on Poverty: Documenting impacts through Ripple Effect Mapping, Community Development, 47:3,

385-402, DOI:10.1080/15575330.2016.1167099

23

Table 3: REM Workshop 1: Self-reported Impacts of Opening Doors (n=4):

Ripple 1 (Specific actions taken by Graduates )

Ripple 2 (Who benefitted and how?)

Ripple 3 (How has the community changed?

Illustrative Quotes

Championed Broadband access for Seniors Program

Improved access to computers for older residents

Increased respect for elders

Increased confidence of community members

Shared stories

Increased networks and connections to address social isolation

Broadened viewpoints in the community about social exclusion

Increased participation in volunteering

Increased lateral thinking about acting on social inclusion

The biggest action which I did - I stopped thinking about myself. I thought what I can do for others? Male Graduate 2013 It gave them somewhere to share their stories. It gave them somewhere to feel non-judged, supported…it was very respectful - everyone's got their own experience and everyone has got an equal right to share their experience, or not share if they choose not to. Female Graduate 2011

Chaired RDNS Community Participant Advisory Committee

Advocated for greater community participant involvement to RDNS

Approached Vision Australia for keyboard enlargements

Improved access to computers for visually impaired residents

Gave presentations on diabetes self-management for Diabetes Australia Victoria Branch

Improved knowledge about diabetes self- management amongst local church group members

Established parent support groups

Increased connections amongst parents who may be at risk of social isolation

Recommended other community members to apply for Opening Doors

Increased opportunities for community members to reduce social isolation

24

Table 4: REM Workshop 2: Self-reported Impacts of Opening Doors (n=3)

Ripple 1 (Specific actions taken by Graduates )

Ripple 2 (Who benefitted and how?)

Ripple 3 (How has the community changed?

Illustrative Quotes

Established a committee to support the delivery of the Black Dog Community Art Exhibition; supporting people with mental ill-health (anxiety and depression)

Inspired artists who participated in The Black Dog to develop an Arts and Culture Collective

Created a questionnaire asking the community about peoples’ reactions to anxiety and depression; generating statements to produce a book for the broader community, schools and nursing homes

Applied for a community development position in a health organisation to support people with chronic mental illness

Empowered and increased self-esteem of people with anxiety and depression to be heard, informed and have a voice

Empowered and inspired people with anxiety and depression through art

Reduced stigma for people with mental ill-health

Built leadership capacity, empowering people living with anxiety and depression

Increased opportunities to employment for people living with anxiety and depression

Opening Doors was a safe environment. They nurtured my confidence and my strengths. The strengths-based teaching was really significant for me and I've used that with my clients. I do work on that way. It's had a real ripple effect on my clients. So it's built their confidence. I didn't know I could publicly speak until I… was on the course. Female Graduate 2015 The idea of the welcome dinner project is to bring people together…a lot of people to Australia, migrants, refugees, anyone who comes, and to connect them with established Australians over a dinner in local homes. Female Graduate 2015

Established connections with the existing Welcome Dinner Project

Organised a Welcome Dinner in Narre Warren for 25 migrants and refugees

Connected newly arrived migrants and refugees with established Australians

Increased connections between migrants and refugees and Australians in the local community

Increased awareness of local community leaders about the value of Welcome Dinners as a way to promote social inclusion

Established partnership between Women’s Health, Benevolence Australia and Migrant Information Centre to understand what

Increased awareness of health services about Muslim women’s experiences of care

Built knowledge about social inclusion at a local level and developed practical skills to apply in promoting social inclusion

25

Muslim women experienced when engaging with health services

Table 5: REM Workshop 3: Self-reported Impacts of Opening Doors (n=3)

Ripple 1 (Specific actions taken by Graduates )

Ripple 2 (Who benefitted and how?)

Ripple 3 (How has the community changed?

Illustrative Quotes

Supported the development of a three year plan for Benevolence Australia, including the creation of an outdoor garden space for the whole community

Applied for funding to grow fresh produce in outdoor garden at Benevolence Australia

Built self-esteem of Opening Doors participants to establish new connections with Benevolence and the broader Muslim community

Supported Opening Doors graduates to volunteer at Benevolence

Brought people together through outdoor garden

Increased volunteering

Supported the advocacy work of Benevolence Australia to increase social inclusion amongst members of the Muslim community

I basically had quite clear project ideas. So the project was really about creating a three year plan for Benevolence in terms of the garden space out the back and really making it more inviting. Female Graduate 2015

Opening Doors really helped me to build my self-esteem, because my self-esteem had gone really - become really low... It really helped me to just know that there were other people who were like me. I wasn't on my own and they were really, really kind, gentle people who were willing to nurture me. Female Graduate 2015 I have no doubt that the increase in self-esteem and confidence is absolutely a take home on an individual level. Opening Doors is very, very powerful for that. Also in terms of, not so much confidence, but having to harness your motivation to then go on and do something with it. Female Graduate 2015

Created a neighbourhood information resource about local services and businesses

Increased knowledge of new families about how to access local services and businesses

Increasing local community know-how about access local community services and businesses

Increased feeling of belonging in the local community

26

Table 6: REM Workshop 4: Self-reported Impacts of Opening Doors (n=3)

Ripple 1 (Specific actions taken by Graduates )

Ripple 2 (Who benefitted and how?)

Ripple 3 (How has the community changed?

Illustrative Quotes

Facilitated Indian Senior Citizens Association to connect and work more collaboratively to promote social inclusion

Applied for and was elected President of the Indian Senior Citizens Association

Identified and accessed facilities within the local community for project work

Established ‘MyStory’ in the Indian Senior Citizens Association – sharing the most memorable part of our lives

Connecting with other Seniors Organisations – Chinese, Greek, African

Connecting with the Aboriginal Community to involve them in the annual Indian Senior Citizens Association multicultural day celebration

Participated in fellow graduate’s program RightClick- a program to bring seniors and school children together to teach seniors how to be technology savvy

Increased awareness amongst Indian Senior Citizens about social isolation and ways to promote social inclusion

Increased knowledge and connections between the Indian Senior Citizens Association and other senior organisations from other cultures

Built connections amongst a community of like-minded people who are passionate about social inclusion

Built networks and connections between local community organisations

Creating opportunities to connect Senior organisations from all cultures

Increased knowledge and skills

Opening Doors gave me a platform to work towards the goals that I wanted to achieve in my life. That was to create happiness amongst people, to join them together and probably fill the gap in their life that they had formed as a result of either isolation or their circumstances they were in…’ Male Graduate 2015

Provided on-sea experiences in leadership and team-building through the Grow Sail Project

Worked with refugees, asylum seekers and other at risk groups through the Grow Sail Project

Increased self-esteem and confidence within refugee, asylum seeker and other at risk groups

Refugees and asylum seekers are now better connected to their communities, and have a deeper understanding of their leadership and collaborative abilities

27

Established the Stop, Unwind Sing program (an unstructured community choir with a focus on social connection and fun)

Increased local community health, well-being and connections through singing

Leading and creating change within local communities through singing (choirs)

Table 7: REM Workshop 5: Self-reported Impacts of Opening Doors acts of Opening Doors (n=6)

Ripple 1 (Specific actions taken by Graduates )

Ripple 2 (Who benefitted and how?)

Ripple 3 (How has the community changed?

Illustrative Quotes

Developed a community-based program to help ex-offenders navigate their return into the community after jail

Built platform to view life through an asset-based community development approach

Provided a platform for a disadvantaged community to have a voice

Increased volunteering

Increased local connections

Increased awareness about how stigma effects recidivism rates amongst ex-offenders

…the project was basically around creating more awareness for the general community around disability issues. I would go and talk to different groups, different demographics, different ethnicities, the whole lot, kids, adults, whoever would listen basically about my life story, and how to be an inclusive group rather than a segregated group... I think a sense of respect and appreciation of what each person brings and looking at what is possible, that there is a lot of potential in your ideas and what can be made to happen. Female Graduate 2012

Created awareness in the community around disability issues via giving talks and involvement in public relations

Running public campaigns and testing public transport for accessibility issues

Contributed to formulation of Doncaster Public Transport Policy re; access and equity issues

Advocated for access and equity for people with disability

Improved access to public transport and provided a voice for those who have experienced accessibility challenges

Increased awareness about disability access and equity issue amongst local organisations (local government, etc.)

Increased awareness about disability access and equity issue amongst the general community

28

Table 8: REM Workshop 6: Self-reported Impacts of Opening Doors (n=4)

Ripple 1 (Specific actions taken by Graduates )

Ripple 2 (Who benefitted and how?)

Ripple 3 (How has the community changed?

Illustrative Quotes

Facilitated the running of a depression and anxiety support group

Members of group felt better supported and understood, with particular reference to their mental health

Members of group feel more empowered in the community, and community organisations have a deeper understanding of mental illness

Some of the ripple effects have been about educating the other people in the community. We've got parents in tears. Parents have been crying. Because there's finally somewhere safe for them to go…So the parents are happy and now we've got funding so we've prevented some of the financial barriers. But the flow on effects, the staff are all vying to actually work for the event [TeenDinners] because they're all seeing that it's quite a powerful event to be a part of. Female Graduate 2015

Established the ‘Pathways for Carers’ project in Maroondah, The Yarra Ranges and Manningham

Participated on Boards in a voluntary capacity (Association for Children with Disabilities; Interchange Outer East; Maroondah Disability Advisory Committee)

Established Parent support Group for parents of children with autism at schools

Established TeenDinners for parents of children with Autism Spectrum Disorder (ASD)

Created a space (via TeenDinners) for parents of children with Autism Spectrum Disorder (ASD) to feel safe, be heard, become informed and have a voice

Created an opportunity (via WALKS) for carers, service providers to learn about support and services available

Carers with anxiety and depression feeling more comfortable to see service providers

Increased knowledge and connections (via TeenDinners) between service providers and parent of children with ASD and carer issues

Informed key Disability organisations about the needs of parents of children with (ASD) and carers in general

Learned how to write business plans to obtain sponsorships / grants

Increased knowledge, understanding and acceptance of parent of children with ASD and carer issues

Empowered parents of children with Autism Spectrum Disorder (ASD) and carers

Gave parents of children with Autism Spectrum Disorder (ASD) hope

Increased volunteering in the community

Existing and emerging community leaders feel empowered, educated and connected

Established The Youth Collaboration Network, enabling the largest and most active organisations who engage with youth to share their learnings and experiences

Facilitated non-government organisations (St Vincent De Paul Society, Salvation Army and Jesuit Social Services) to collaborate to support youth leaders to work together

Created a culture of leadership which encourages youth to participate

Increased collaboration amongst organisation to support youth leaders

29

Table 9: REM Workshop 6: Self-reported Impacts of Opening Doors (n=3)

Ripple 1 (Specific actions taken by Graduates )

Ripple 2 (Who benefitted and how?)

Ripple 3 (How has the community changed?

Illustrative Quotes

Established a support group (Transfamily) for parents of transgender people and other family members

Wrote book for parents of transgender people and other family members

Spoke to Monash University trainee Doctors about Transfamily and the issues facing parents of trans children

Connected to Benevolence Australia

Improved awareness and advocated for parents of transgender people and other family members

Helped to remove stigma among people with mental ill-health

Helped clients to better recognise the own needs

Increased knowledge and acceptance about transgender people among next generation of doctors

Improved awareness and understanding about transgender people in organisations (eg DonCare, Benevolence Australia)

Increased awareness and contributed to building a culture of acceptance among the general community for transgender people

Increased advocacy for the rights of transgender people

Empowering others to becomes advocates in their world

Increased connections amongst and between communities

I found that those young trainee doctors had very little knowledge about transgender people and how they might treat them if they came into their office…I believe this is a ripple effect. Those young doctors will now go out into their world as they're educated and … they're certainly more aware and they knew nothing. They knew nothing about the families. Female 2012 Graduate

Established ‘Pleasures of the pen’ project to connect older people

Established ‘Poetry at the lodge’ (Aged Care Facility)

Improved written communication skills of older people in Aged care Facilities

Connected community members at risk of isolation via a shared love of the written word

Increased communication amongst older people

Established a storytelling workshop for people who have experienced harm from gambling addiction

Produced a book which shares the stories of people who have experienced harm from gambling addiction / problem gambling

Empowering people struggling with gambling addiction to find their voice on the journey to recovery and to become advocates for change

An increase in the depth of understanding of the risks of gambling, as well as the challenges and stigmas faced by those who have experienced harm from pokies (EGMs)

30

Opening Doors Steering Committee perspectives The Ripple Effect Mapping workshop with the Opening Doors Steering Committee also revealed

insights into the impacts of Opening Doors. Table 10 provides a summary of comments made by

Steering Committee members when asked about the impacts of Opening Doors.

Table 10: Summary of Opening Doors Steering Committee Members Reported Impacts of Opening

Doors

Increased general awareness about Opening Doors in the sector via greater recognition and media

Unintended consequences – e.g. an Opening Doors graduate got married, and she started teaching

Opening Doors is broader than projects – i.e. projects are a bi-product and part of the process

Opening Doors is a learning program

Broader support for Opening Doors e.g. police officer attended graduation

Opening Doors Project are what we see - just tip of the iceberg - ripple effects are invisible intangible - changing relationships and perceptions

Opening Doors has provided opportunities for community members to embrace

Opening Doors involves a long term commitment, as often graduates do things after the program. i.e. the trajectory of change or what Opening Doors leads to/facilitates or effects - is not linear

Opening Doors Contextual Factors: Enablers/barriers

This section provides findings from the Ripple Effect Mapping workshops, with regard to the

enablers and barriers facing Opening Doors and its impacts; firstly from the Opening Doors

graduates, and then the Steering Committee.

Opening Doors Graduates perspectives Opening Doors graduates reported multiple enablers (Table 11) and barriers (Table 12) facing

Opening Doors and its impacts that can be summarised at an: individual, program, organisational,

and systems level.

Table 11: Summary of Enablers facing Opening Doors

Level Enablers Illustrative Quotes

Individual

The Coordinator qualities, including

Competent; benevolent; non-judgmental; networker; honest; open; goodwill; reliable; accepting; personable

Having lived experience

Having been an Opening Doors participant

I think credit where credit is due. The Coordinator has been quite outstanding in terms of being very responsive to everyone's differences and what they're bringing to the table. He's been very understanding... I think that he does exude the fact that he's a warm, open, engaging person. He does want people to flourish. He does want to motivate people. They're things that are definitely good. Female Graduate 2012 … the asset-based community development approach. For me, it was something that changed my opinion about lots of things in my life. Female Graduate 2015

Organisational Opening Doors structure (Live-in retreat, followed by workshops); Content – Global Leadership Foundation Speaker;

The ABCD community development approach;

Auspiced and supported by a Steering Committee with committed members and not only one organisation

Systems

Opening Doors connected to multiple organisations and networks

31

Table 12: Summary of Barriers facing Opening Doors

Level Barriers Illustrative Quotes

Individual

Time limitations, limited Coordinator capacity to support all past & present graduates

Dependency on Coordinator for graduates knowledge, transfer and exchange; volunteer fatigue

Ripples are often not visible

People do become very dependent on [the Coordinator] and the boundaries between doing your job and being a personal friend and if he becomes a personal friend of 30 people per year, so what's that since 2012 to '16? As much as he's a warm and generous person, he probably doesn't have quite enough room for everyone in his life. I think that because everything is put out on the table you think that you have that bond is to some degree created. Female Graduate 2012 The barrier I'm thinking of is for people who work full time whose work wouldn't support it. Female Graduate 2015 We know Opening Doors is fantastic and we know it makes a huge difference, but if maybe business understand the value that they could get from it as a reciprocal relationship they probably would jump all over it. Female Graduate 2012

Organisational Opening Doors needs to have a balance between structure, activities and reflections

Variable organisational support for participants who work full-time

Systems

Limited evidence of the explicit monetary value of Opening Doors exists

Opening Doors is not accredited nor aligned with other Leadership programs

Opening Doors graduates also made multiple suggestions to sustain Opening Doors and its effects

that can be summarised into two levels:

Table 13: Summary of Graduates Strategies to Sustain Opening Doors and its effects

Level Illustrative Quotes

Individual level

Need to have a shared Coordinator position

Need to provide social and skill-based opportunities for graduates (virtual or face to face) for graduates to get together

Need to provide a follow-up one year after the program for graduates

Need to have structured feedback opportunities for graduates

Opening Doors alumni could support more current participants

Keeping people engaged with it. Female Graduate 2015 There's almost room for a mentoring role to some degree… that would work really well actually: someone from a past previous year, because you have an inkling of what participants would be thinking. Female Graduate 2015

Program level

Need funding to support graduates to get together

Need to provide opportunities (via website or face to face) for graduates to share ideas of all past and present Opening Doors participants

need to establish networking opportunities to sustain momentum

Share the growth and keep the motivation because even today simply being in the same room as [Graduate] I've gone oh yeah I remember that feeling. Oh yeah, yeah, yeah Female Graduate 2012 I think ongoing networking opportunities would be brilliant. Maybe quarterly events, dinners or having a guest speaker or something for everybody who's done Opening Doors

32

need to provide evaluative and reflective spaces for graduates

Female Graduate 2012 This is where the Coordinator needs more support I think. You could almost have a Facebook group that supports each other, like an alumni group. Social media. It would be great to have more contact with people. As [Graduate] said, just us being in the room together is - I'm feeling really inspired and enthusiastic and it's just one person. Imagine if we had 10 people. Female Graduate 2015

33

Opening Doors Steering Committee perspectives The Opening Doors Steering committee members mentioned the following enablers and barriers

facing Opening Doors and its effects:

Table 14: Summary of Enablers and Barriers facing Opening Doors from Committee Members

Enablers Barriers

Opening Doors is a communal investment

Opening Doors is designed as leadership program, not project-based –i.e. it does not matter if the project does not get off ground – it is the lived experience of Opening Doors that matters

The people and their passion

The community development frame

Opening Doors committee is resilient i.e. responsive to almost going under

Opening Doors is not in a box

Opening Doors is targeted - it is not just aimed at vulnerable people, but to all who are dedicated to increasing social inclusion

Peer support for Opening Doors program Senior (e.g. Police officer who came to support his officer who did the Opening Doors program)

Opening Doors is not their (e.g. local government) idea

Opening Doors has no official status nor ongoing funding

Opening Doors does not fit in a box –thus hard to secure funding - a paradox

Need engagement strategies to create a wider net out there to inform people and organisations about Opening Doors

Need to resource Opening Doors graduate voices

Need to explore SociaI Inclusion Committee providing a structure for Opening Doors

Need authorising environment for Opening Doors to demonstrate its value

Opening Doors Steering committee members also mentioned multiple strategies that are key to

sustaining Opening Doors and its effects, including that Opening Doors requires:

a dedicated, committed ongoing Coordinator to inspire and support participants and

graduates

a Steering Committee that auspices it as a network (not one organisation owning it) and

supports its members coming and going – as ongoing commitment

the Program to be seen as a vocation and not a job

a high profile

graduates lived experiences and go-get nature

sustainability is about the values, the people, the resources and the outcomes of Opening

Doors.

34

Synthesis and Discussion of Evaluation Findings

This section synthesises and discusses the evaluation under four headings:

1. Impact of Opening Doors;

2. Benefits to Opening Doors Participants;

3. Critical ingredients for success;

4. Reflections on evaluation methodology.

Impact of Opening Doors This section initially synthesizes evaluation findings in relation the impact of Opening Doors in the

community and then on reducing social isolation.

Please note: given that only 26 graduates (22% of all graduates) participated in the REM workshops,

the actions and ripples are not an exhaustive list but illustrative of the spectrum of ripples or impacts

and value of Opening Doors.

Please note: the Ripple Effect Mapping workshops were not project specific, but designed to capture the diversity of graduate experiences and effects of participating in Opening Doors.

Impact in the Community The Ripple Effect Mapping workshops with the 26 Opening Doors graduates reveal that since 2009

Opening Doors has successfully recruited and supported a diverse group of local community leaders

to transform participants’ ideas into projects to create more socially inclusive communities.

Opening Doors graduates have taken multiple actions to create more socially inclusive communities

that can be summarised into three levels:

Individual level: established multiple projects; championed access issues; established connections, accessed facilities; advocated for Opening Doors; applied for and obtained employment positions;

Organisational level: established structures, committees and partnerships; contributed to strategic planning processes; took up advisory roles on disability organisations; took up leadership roles in senior citizen organisations; supported strategic plans; contributed to disability access policies; and

Community level: applied for funding to conduct projects; established and facilitated support groups; developed resources (books) for communities; connected community groups.

Opening Doors has benefitted multiple individuals and communities, including: older people; Indian

senior citizens; people with visual impairment; people with chronic disease such as diabetes; parents

in general; parents with children with Autism Spectrum Disorder (ASD); people with anxiety,

depression and mental illness; newly arrived migrants; refugee children; transgender people;

parents of transgender families; people with gambling addictions; ex-offenders; people with a

disability; and carers of people with health conditions or other social disadvantage.

The benefits for communities can be clustered into five categories, including:

Access - Increased access to technology ;

Knowledge- increased knowledge of health conditions, local services; support systems for carers;

Capabilities - increased self-esteem; increased confidence;

35

Advocacy - increased spaces to be heard and have a voice for carers, parents of children with ASD, people with anxiety and depression, parents of transgender families;

Connections - increased connections between newly arrived migrants and local Australians.

Opening Doors has also benefitted health professionals (e.g. trainee doctors); health service

organisations (e.g. RDNS); cultural specific organisations (e.g. Indian Senior Citizens Association);

Residential Aged Care organisations (e.g. DonCare); non-government organisations (e.g. St Vincent’s

De Paul; Jesuit Social Services); local government organisations (e.g. Maroondah City Council).

The benefits for those professionals and organisations can be clustered into two main categories:

Knowledge – increased knowledge of issues facing: carers; parents of children with ASD; ex-offenders; people with a mental illness, people with disabilities, and parents of transgender families.

Connections & Collaborations - increased connections and collaborations between Senior Citizen Associations; non-government organisations to support youth leadership.

Opening Doors is contributing to changes in the broader community that can be clustered into eight

domains:

1. Increasing the awareness and knowledge of factors that contribute to social isolation, social

exclusion and those factors that promote social inclusion

2. Increasing the quality and number of relationships, connections, networks and

collaborations

3. Increasing ideas, viewpoints, mindsets, and ways of thinking about social inclusion using

asset-based community development approaches

4. Increasing the leadership capabilities - increased empowerment, confidence and advocacy

5. Increasing the level of participation and engagement of marginalized local communities

6. Increasing the level of respect and acceptance of people who are marginalised and socially

excluded from society

7. Increasing the level of belonging and decreasing the level of loneliness of marginalized

communities

8. Increasing the level of volunteering, generosity, goodwill and benevolence - giving back to

others

Overall these evaluation findings confirm previous evaluations of Opening Doors by Held (2011)

which identified two key outcomes of Opening Doors, namely: increased participant’s leadership

knowledge, skills and confidence to engage with communities; and establishment of community-

based projects to strengthen community connectedness and reduce social isolation in local

neighbourhoods across the inner east catchment of Melbourne.19

The Ripple Effect Mapping evaluation workshops further revealed that Opening Doors is contributing

to building community capacity as demonstrated by:

development local leadership; improving community participation; improving local

assessment of problems and assets; improving local resource mobilization; strengthening

local community connections/collaborations/partnerships; building local structures; and

building local project capabilities.

19 Held, R. (2011). Opening Doors: A community leadership program for social inclusion. Evaluation Report. Inner East Primary care Partnership. http://iepcp.org.au/sites/default/files/Report%20IESII%20Evaluation%20Final_0.pdf

36

The Ripple Effect Mapping evaluation workshops have further revealed that Opening Doors is

contributing to building community capacity (defined as the community groups ability to define,

assess, analyse and act on health (or any other) concern of importance to their members) as

demonstrated by:

development of local leadership;

improving community participation

improving local assessment of problems and assets;

improving local resource mobilization

strengthening local community connections/collaborations/partnerships

building local structures

building local project capabilities.20

Future evaluations of Opening Doors could utilise recognized community capacity building

dimensions as an ongoing monitoring and evaluation framework.

The Ripple Effect Mapping workshops also demonstrated that Opening Doors is building social

capital, by building trust amongst its participants and enhancing their willingness to act upon ideas,

and by increasing their self-empowerment to then do for others. Putnam (1995) defined social

capital as - features of social life – networks, norms and trust – that enable participants to act more

effectively to pursue shared objectives.21 Social capital as ‘collective action’ has three major

components or pre-conditions: trust, social networks and civic engagement. The workshops also

demonstrated how the Opening Doors model and approach builds multiple types of trust between

its participants and the Coordinator. Opening Doors utilises horizontal networks as a vehicle for

trust, as information about the trustworthiness of other people becomes spread.

The findings from the Ripple Effect Mapping workshops and the design of Opening Doors (i.e. to

build leaders in local communities with the knowledge, skills, resources and networks) also resonate

with the work on Community Capitals.22 Capital has been described as any type of resource capable

of producing additional resources. When those resources or assets are invested, they become

capital. Community capitals are all of the things in a community that have the potential to be a

resource that can be invested, saved, or used up.23 Seven types of community capital have been

identified that can be used to gauge how community resources are being used: Natural Capital;

Cultural Capital; Human Capital; Social Capital; Political Capital; Built Capital; and Financial Capital.

Figure 1 provides a visual representation and Appendix D provides further details. The Community

Capitals framework has been used as a framework to plan, implement and evaluate community

development programs, while also using Ripple Effect Mapping to capture the voices of community

members.

Future evaluations of Opening Doors could utilise the Community Capitals framework as an ongoing

planning, monitoring and evaluation framework.

20 Gibbon, M; Labonte, R and Laverick, G. (2002). Evaluating community capacity. Health and Social Care in the Community. 10(6):485-491. 21 Putnam, RD . (1995). Bowling alone: America’s declining social capital. Journal of Democracy, 1: 65-78. 22 Jacobs, C. (2011). Measuring success in communities: Understanding the Community Capitals Framework. Extension Extra, 16005.

Retrieved from http://agron-www.agron.iastate.edu/Courses/agron515/CapitalsExtension%20Extra.pdf 23 Emery, M., S. Fey, C. Flora. 2005, Using Community Capitals to Develop Assets for Positive Community Change. Community Capitals

Framework: Research, Evaluation and Practice Conference, Ames, Iowa.

37

Figure 1: Visual Representation of the Seven Types of Community Capital24

Impact on Reducing Social Isolation Based on the collective impact of the Opening Doors graduates projects, actions and ripples in the

community, one could surmise that Opening Doors is contributing to reducing social isolation.

However, caution is required if making a ‘cause and effect’ judgment. As a community development

and empowerment program, Opening Doors is designed to influence multiple social phenomena -

social exclusion; social inclusion; social isolation; social cohesion; social capital; community

participation; marginalisation and community engagement - all of which have conceptual,

definitional and measurement issues.

In the next section we reflect briefly upon the concepts and definitions of social isolation; social

capital; social exclusion and social inclusion, and then the challenges in measuring social isolation.

While social isolation is described as a deprivation of social connectedness, numerous definitions of

social isolation exist due to diverse sociological and psychological theories. Social isolation

definitions have been categorised into those that seek to employ objective measures (number of

relationships, social interactions, extent of networks) and subjective measures (quality of

interactions, feelings of loneliness).25 26 Zavaleta et al’s definition of social isolation reflects the

multi-dimensional nature of social isolation: ‘the inadequate quality and quantity of social relations

with other people at the different levels where human interaction takes place (individual, group,

community and larger social environment’).27

Social capital is also a key Opening Doors concept which has multiple definitions. Social capital refers

to social networks that are based on trust and reciprocity that enable people to collectively resolve

24 Jacobs, C. (2011). Measuring success in communities: Understanding the Community Capitals. Framework. Extension Extra, 16005. Retrieved from http://agron-www.agron.iastate.edu/Courses/agron515/CapitalsExtension%20Extra.pdf 25 Cattan, M., White, M., Bond, J., & Learmouth, A. (2005). Preventing social isolation and loneliness among older people: A systematic

review of health promotion interventions. Ageing & Society, 25, 41–67. 26 Findlay, R., & Cartwright, C. (2002). Social isolation and older people: A literature review. Brisbane: Australasian Centre on Ageing,

University of Queensland. 27 Zavaleta, D., Samuel, K. & Mills, C. (2014). Social isolation: A conceptual and measurement proposal. OPHI Working Papers 67. http://www.ophi.org.uk/social-isolation-a-conceptual-and-measurement-proposal/

38

common problems and achieve common goals.28 Social capital is also usually described as having

three key components; bonding social capital (refers to trusting social connections between

members of a network that build on commonalities and homogeneity in terms of their shared social

identity); bridging social capital (refers to relations of respect across diverse social groups); and

linking social capital (refers to alliances between communities and individuals or groups who are

interacting across explicit, formal or institutionalised authority).29

Social exclusion and inclusion relate to patterns of social integration, one’s living situation and

patterns of everyday social contact, and on well-being and feelings of loneliness. Social exclusion has

also been defined in a multitude of ways. For example, the European Foundation for the

Improvement of Living and Working Conditions defined social exclusion as ‘the process through

which individuals or groups are wholly or partially excluded from full participation in the society

within which they live’.30 While theories of social exclusion provide insights for understanding the

level of connectivity of a person, Silver (2007) argues that one cannot separate the exclusion of a

group from the evaluation of the quality of social relationships.31 Social Inclusion is also a complex,

dynamic and multidimensional process about participation, equal opportunity, and empowerment

and has been defined as having the resources, opportunities and capabilities to learn, work, engage

and have a voice. 32 33 34

Challenges with defining social isolation, social exclusion, social inclusion, and social capital has led

to the wide use of multiple indicators and proxies for their measurement. Furthermore, while many

interventions seeking to reduce social isolation have been implemented, few evaluations of the

effectiveness, sustainability and long term benefits of interventions exist. Zavaleta et al’s review

identified fifty potential indicators to provide data on social isolation.35 In the literature there are

also multiple measures or indicators of social isolation including: feelings of loneliness; low level of

social contacts; gender; and mobility restrictions. Multiple scales and indexes that measure issues

that intersect with social isolation and social exclusion also exist, that draw from loneliness scales;

social capital indicators; well-being indicators; quality of life indicators and social isolation indicators.

28 Stone, W. (2001). Measuring social capital: Towards a theoretically informed measurement framework for researching social capital in

family and community life Melbourne (Research Paper No. 24). Melbourne: Australian Institute of Family Studies. 29 Putnam, R. D. (2001). Social capital: Measurement and consequences. Isuma: Canadian Journal of Policy Research, 2, 41–51 30 Deakin, N., Davis, A., & Thomas, N. (1995). Public welfare services and social exclusion: The development of consumer oriented initiatives in the European Union. Dublin: European Foundation for the Improvement of Living and Working Conditions 31 Silver, H. (2007). The process of social exclusion: the dynamics of an evolving concept. Rhode Island, Chronic Poverty Research Centre,

Brown University 32 Symth, P.(2010). In or out? Building an Inclusive Nation. www.australiancollaboration.com.au 33 Morrison, Z (2010). On dignity Social inclusion and the politics of recognition. www.public-policy.unimelb.edu.au 34 Australian Social Inclusion Board (2013). Social inclusion in Australia: How Australia is faring. At

http://www.socialinclusion.gov.au/resources/how-australia-is-faring 35 Zavaleta, D., Samuel, K. & Mills, C. (2014). Social isolation: A conceptual and measurement proposal. OPHI Working Papers 67. http://www.ophi.org.uk/social-isolation-a-conceptual-and-measurement-proposal/

39

Zavaleta et al’s review proposed a conceptual framework (two domains: external and internal social isolation), proxy indicators, and instruments for measuring social isolation:

1. External social isolation:

Frequency of social contact

Social network support

Reciprocity and volunteering

2. Internal social isolation

Satisfaction with social relations

Need for relatedness

Feeling of belonging to one’s own neighbourhood

Loneliness

Trust. 36

To provide a definitive assessment of whether Opening Doors is reducing social isolation, a

quantitative evaluation approach using validated instruments would have been required.

Drawing upon the Ripple Effect Mapping workshops, we can speculate that Opening Doors is

contributing to addressing proxy indicators of social isolation as suggested by Zaveleta et al, namely:

external social inclusion (e.g. increasing the frequency of social contact; increasing social network

support; increasing reciprocity and volunteering) and internal social isolation (e.g. increasing

satisfaction with social relations; increased need for relatedness; increased feeling of belonging;

decreased loneliness and increased trust).

Future evaluations of Opening Doors could consider utilising Zavelata et al’s social isolation

conceptual framework and validated instruments to assess whether Opening Doors is reducing social

isolation.

The complexity of the concepts and evaIuation task was also evident right from the beginning of the

Ripple Effect Mapping workshops. As mentioned previously the Ripple Effect Mapping methodology

was designed to capture not only the ‘effects/ripples’ but the experiences of graduates, which the

evaluator explored at the start of both the Graduates and Committee Members Workshops by

providing a scene setting statement (Creating a Culture of Social inclusion) and a framework: values,

symbols, practices and systems, and asking for reflections. As can be seen in Appendix F, overall, the

scene setting activity worked well - with participants providing multiple insights into what a ‘Culture

of Social Inclusion’ looked like and encompassed.

Benefits to Opening Doors Participants The Ripple Effect Mapping workshops have confirmed previous evaluations of Opening Doors that

Opening Doors is building its participants’ leadership capability as demonstrated by:

Supporting and building its participants’ leadership assets and strengths to inspire others to

work with them to create a greater sense of community that is inclusive of age, ability and

cultural diversity;

36 Zavaleta, D., Samuel, K. & Mills, C. (2014). Social isolation: A conceptual and measurement proposal. OPHI Working Papers 67. http://www.ophi.org.uk/social-isolation-a-conceptual-and-measurement-proposal/

40

Strengthening its participants’ relationships, connections and networks to transform ideas

and design, plan and implement projects to increase social inclusion and address social

inclusion issues in their local community;

Supporting its participants’ personal leadership knowledge, skills and resources to use and

apply asset/strength-based community development approaches to address social isolation

that people experience.

Given that considerable time, money and resources has been invested into Opening Doors as a

Community Leadership Program for social inclusion, a brief reflection with regard to the

conceptualisation of leadership development programs and the link to social capital and community

leadership is appropriate here. Opening Doors is a community leadership program currently targeted

at community leaders. However, differing types of leadership development programs exist, due to

differences between a ‘leader’ and ‘leadership development’. Traditional conceptualisations of

leadership (e.g. transformational leadership) emphasise individual leaders – therefore, training

programs that seek to improve individual skills and abilities are referred to leader development or

human capital development. Whereas, programs that develop collective leadership capacity of

groups, organisations or community should be referred to as leadership development programs or

Social Capital development.37

The workshops have revealed that Opening Doors is developing both the skills and capacity of

individuals (leader or human capital development) and the collective capacity of groups and the

community (leadership or social capital development). To further develop and sustain the capacity

of Opening Doors to create more socially inclusive communities, a continued focus upon

leader/human capital and community/social capital development is required.

The Ripple Effect Mapping workshops also revealed that Opening Doors is facilitating and inspiring

volunteerism among its graduates and beyond - as it has produced benefits for the volunteers

themselves, the recipients of their services, the organisations for which they work or engage with,

the community, and the broader society.38 Given that Opening Doors is a free program, the reliance

on volunteers is increasing, the recruitment and retention of volunteers is a continual challenge, and

there is a steady decrease in weekly voluntary contributions of time and efforts. Opening Doors

needs to reflect upon strategies to optimising the recruitment and retention of its volunteer

community members.

37 Day, D. V. (2000). Leadership development: A review in context. Leadership Quarterly, 11(4), 581(33). 38 Stukas, AA; Hoye, R; Nicholson, M; Brwon, KM; Aisbett, L (2016). Motivations to volunteer and their associations with volunteers well-

being. Non-profit and voluntary Sector Quarterly, 45(1): 112-132.

41

Critical ingredients for success The Ripple Effect Mapping workshops with both the graduates and Steering Committee members

revealed the following key ingredients for the success of Opening Doors:

A Coordinator:

Who has core relational qualities (benevolent; non-judgmental; honest; open; goodwill; reliable; accepting and personable)

Who has connections to multiple organisations and networks and know-how

Who has lived experiences in relation to having been an Opening Doors participant

Who has experience putting into practice the Asset-Based Community Development (ABCD) approach to community development

A Program:

that is free and based upon the principle of volunteerism, and open to local community members with a combination of self- and other-oriented drivers for taking action on social inclusion

that has a structure, that includes an initial live-in retreat (providing a safe environment for building relationships), followed by interactive workshops with activities and dedicated spaces and time for reflection; and ending with a graduation that a showcases a graduates actions and impact of their community-based projects

that has content that is pragmatic and practical and includes core resources for building skills and capabilities in: leadership; asset-based community development (ABCD); appreciative enquiry; team building; project development; events and media management. Opening Doors graduates overwhelmingly valued and commented that the Asset-Based Community Development (ABCD) approach to community development was a key enabler to the success of Opening Doors, identifying, developing and mobilising existing strengths (assets) in the community, building community networks, and transforming ideas, mindsets, assumptions and expectations into community-based projects to address local needs.

that is auspiced and supported by a Steering Committee comprised of a network of organisations

that is not project-based but designed to build community leaders and leadership

that is supported and authorised by a wide spectrum of health, human and social service organisations

A Post-Program Strategy:

that supports its graduates with social and skill-based opportunities (virtual, face to face,

forums) to maintain and establish new connections, share new knowledge and ideas, and

to reflect with other graduates.

Reflections on Evaluation Methodology This section briefly reflects upon the Ripple Effect Mapping evaluation methodology, and the

Opening Doors Program Logic Model developed in February 2016.

Ripple Effect Mapping The Ripple Effect Mapping workshops have generated rich qualitative evidence about the complexity

of Opening Doors and the actions and effects resulting from its graduates. The workshops revealed

how the evaluation methodology has an effect on the group dynamics. Graduates generated energy

as they participated in the workshops - often assisting them envisioning their personal and

community impact, while sharing and collecting words and relationships for sharing the self-

42

reported impacts of their Opening Doors projects. The Ripple Effect Mapping workshop accentuated

the graduates relationships and connections developed while participating in the Opening Doors. As

a participatory evaluation method, the Ripple Effect Mapping workshops assisted the graduates to

express their voices and show how their ideas were transformed into community-based projects

that contribute to creating more socially inclusive communities. The workshops expanded graduates

range of thinking about how far their impact was reaching and the difference they were making.

Ripple Effect Mapping workshops also provided graduates with further opportunities to connect

with other past graduates. More specifically the workshops contributed to building social capital

among Opening Doors graduates - bonding capital (re-connecting with known graduates) and

bridging capital (connecting with new graduates) to put ideas into action.

An initial Ripple Effect Map was produced in May 2015 based upon a preliminary analysis of the

workshop transcripts (Appendix E). However, given the complexity of Opening Doors the draft Ripple

Effect Map has not been used formally to illustrate the Opening Doors Ripple effects – rather, the

Ripple Effect Mapping Template (Table 3 - Table 9) have been used. While the Ripple Effect Mapping

workshops have not generated individual, community or project specific Visual Ripple Maps, the

information generated from the workshops can inform future evaluations and questions to further

the work of Opening Doors to create more socially inclusive communities.

While the Ripple Effect Mapping methodology has multiple benefits, we acknowledge that a

limitation is the risk of bias in participant selection, resulting in selective or incomplete information

about outcomes or consequences - both positive or negative. Given the rich, frank and honest views

expressed by all graduates in the workshops about Opening Doors, we consider the methodology as

successful in capturing the diversity in graduate experiences. We also acknowledge that given that

the size and composition of the graduate workshops were beyond the sphere of control of the

evaluator, year specific or project-specific Ripple Effect Mapping workshops may have generated

more in-depth information about the specific actions and impacts resulting from the projects.

Opening Doors Program Logic Model Reflecting upon the Opening Doors Program Logic Model developed in February 2016 (Appendix B),

the Ripple Effect Mapping workshops reveals and provides evidence that:

The hypothesised program logic underpinning Opening Doors is confirmed. The hypothesised contextual factors were confirmed, and expanded upon and remain key issues needing to be recognised including: the importance of the Coordinator core qualities (has lived experience, competent, benevolent, non-judgemental, honest, trustworthy; personable); the networked-based nature of Opening Doors; the dependency by graduates upon the Coordinator; limited monetary value of Opening Doors; and lack of accreditation and/or alignment to other leadership programs.

The hypothesised project inputs necessary for Opening Doors to be implemented were also confirmed and expanded including: asset-based community development approach; importance of Steering Committee leadership and commitment; Coordinator with lived experience, commitment and core qualities; and the endorsement by the social, health and community services sectors and agencies.

The hypothesised activities were also reinforced. The Ripple Effect Mapping workshops revealed how the activities were interdependent (i.e. not mutually exclusive) in other words - it is difficult to identify any causal or direct/linear links between activities (program balanced with structure, activities and reflection space; program links to social, health and community service networks; and the importance of having an asset-based community

43

development program) and actual outputs. It is the cumulative impact of the activities that initially creates the SPLASH (outputs) and then the RIPPLE effects (outcomes).

Opening Doors has created a SPLASH; i.e. outputs/actions at three levels: individual (established connections, accessed facilities; advocated for Opening Doors; applied for/obtained jobs); organisational (established structures; contributed to strategic planning processes; taken up advisory roles); and community (taken up leadership roles; developed resources, e.g. books, Transfamily, gambling) level.

The short evaluation time frame limits our ability to comment on intermediate and long-

term Outcomes achieved so far. However, the workshops revealed that Opening Doors is creating multiple transformative and incremental ripples that can be summarised into self-oriented ripples (increased awareness, ideas, opportunities, networks, confidence, self-esteem, empowerment, knowledge, skills; voice, community connections; shift to asset/strength-based thinking, being, and doing) and others-oriented ripples (increased doing for others, awareness of marginalised communities; supported others to share stories; developed/supported/stimulated others to participate social inclusion initiatives (e.g. Teen Dinners; Transfamily, U3A); empowered staff, parents, carers, clients etc; created authorising environment within organisations for social inclusion).

Key Emerging Propositions To optimise the investment and sustainability of Opening Doors and its ripple effects, the following

propositions could be considered:

Opening Doors Program leadership capacity 1. A shared Opening Doors Coordinator model could be considered to enable a shared

workload, to build shared trust and a supportive platform to support past and current

Opening Doors Graduates social inclusion leadership;

2. An Opening Doors alumni mentoring model could be considered to formally support the

Coordinator to support current participants and to reduce the current dependency upon the

Coordinators for ongoing support;

3. The role of the Opening Doors Steering group as a formal structure to support the Opening

Doors Coordinator and the program’s ripple effects could be further explored;

4. The role and contribution of Link Health and Community Service as a formal structure to

support the Opening Doors Coordinator could be further explored.

Opening Doors Program Profile 5. Accrediting and aligning Opening Doors with other Leadership programs (e.g. Global

Leadership Foundation) could be explored as a way of further providing an explicit public

profile for Opening Doors;

6. Given that the local community focused approach of Opening Doors was valued by all

graduates, promoting the local ripple effects to a broader health, social and community

service base could be considered.

Social Inclusion Leadership capacity 7. To sustain the momentum of Opening Doors graduates establishing an Opening Doors

Alumni Community of Interest, facilitated by alumni to provide social connection and skill-

based opportunities (virtual, Face to face, forums) could be explored;

44

8. Investing in an expanded Opening Doors social media platform (webpage, facebook, etc) to

share ideas into action of all past and present Opening Doors participant could be

considered;

9. Given that Opening Doors is a free program relying on volunteers, sustainable strategies to

optimise the recruitment and retention of its volunteer community members need to be

considered.

Opening Doors Monitoring and Evaluation Framework 10. Given that Opening Doors is building community capacity, future evaluations of Opening

Doors could utilise recognized community capacity building dimensions as an ongoing

monitoring and evaluation framework;

11. Given that Opening Doors is investing in resources and assets (Capital) the Community

Capitals framework could be considered as an ongoing planning, monitoring and evaluation

framework;

12. Given that the Ripple Effect Mapping methodology contributes to engaging graduates and

illuminating effects, evaluating project specific changes using Ripple Effect Mapping could be

considered;

13. To assess definitively whether Opening Doors is reducing social isolation, future evaluations

of Opening Doors could consider utilising Zavelata et al’s (2014) social isolation conceptual

framework and validated instruments.39

39 Zavaleta, D., Samuel, K. & Mills, C. (2014). Social isolation: A conceptual and measurement proposal. OPHI Working Papers 67. http://www.ophi.org.uk/social-isolation-a-conceptual-and-measurement-proposal/

45

Appendix A: Ripple Effect Mapping Workshop

Opening Doors Ripple Effect Mapping Workshop Outline (90 mins)

Facilitator: Dr Lucio Naccarella, PhD, The University of Melbourne

1. Introductions (10 mins) (All)

2. Overview of workshop: purpose of the mapping and objectives (5 mins- Lucio)

The University is leading an impact mapping evaluation project to better understand the ripple effects of the

Opening Doors Program upon its participants and beyond.

3. Drivers for Taking Action on Social Inclusion (15 mins) (All)

What were the main reasons for enrolling in Opening Doors?

4. Actions taken to support social inclusions (15 mins) (All)

What actions, if any, have you taken to create more socially inclusive communities

5. Opening Doors Ripple Effects (20 mins) (All)

What changes, if any, have you noticed specifically related to social inclusion?

6. Contextual factors influencing Ripple Effects (20 mins) (All)

What enablers and barriers exist to the Opening Doors program?

What is required to sustain the changes in socially inclusive communities

7. Close: Reflections & Thank you (10 mins) (All)

Thank you for your time and commitment.

For further information please contact: Lucio Naccarella l.naccarella@unimelb.edu.au

46

Appendix B: Opening Doors Program Logic Model

47

Appendix C: Ripple Effect Mapping Reporting Template - Examples Only

From: Welborn et al (2016) Turning the Tide on Poverty: Documenting impacts through Ripple Effect Mapping,

Community Development, 47:3, 385-402, http://dx.doi.org/10.1080/15575330.2016.1167099

48

From: Baker B & Johannes EM (2013) Measuring Social Capital Change using Ripple Mapping, in Calvert M,

Emery M, & Kinsey S (eds), New Directions for Youth Development: Youth Programs as Builders of Social

Capital, New Directions for Youth Development, no. 138.

Appendix D: Community Capitals Framework

49

From: Welborn et al (2016) Turning the Tide on Poverty: Documenting impacts through Ripple Effect Mapping,

Community Development, 47:3, 385-402, http://dx.doi.org/10.1080/15575330.2016.1167099

50

Appendix E: Initial Draft Ripple Effect Map (May, 2015)

51

Appendix F: Reflections on Creating a Culture of Social inclusion

Graduates Comments Committee members Comments

Culture:

it can be a dividing force

need culturally valued dialogue Values

respect

pride

integrity

stand for acceptance

she’ll be right

free from prejudice

shared values

celebrate diversity

multicultural society Practices:

it’s about what you do not say

how you treat people

its history

unspoken mindsets

aware of exclusion

welcome

creating a space/environment

an inviting space

habits

exclusion

engagement

like-minded

shared value

creates a platform; Symbols:

stories/narratives

wheelchair access

CALD awareness

reconcile; Systems

it’s about connections

it’s about networks

Culture:

culture is diverse and based upon love of humanity

equality and equity of race, religion, background and circumstances

people are free to reach their full potential Values:

being appreciative

individuals have a place

commitment to making a difference

acknowledging and celebrating differences/diversity

acceptance of difference

empowering

values that bind

embracing others

understanding differences

valuing lived experience of people in community

spirit of care

missing others - being misses Symbols:

feeling included – being an insider

celebration of diversity

sharing of food Practices:

ideas are explored

focus on strengths

strengths, talents are valued and celebrated

openness

focus on where biggest diff can be made

equity

listening to and hearing stories

making friendships, relationships, knowing neighbours

providing opportunities for voices to be heard

process is important

time to reflect

responsible

equal opportunity – being able to participate Systems:

work supported by government values, policies and funding

dynamic and open systems