Social Sector Trials Evaluation Findings: Establishment Phase July 2011

Social Sector Trials background About the establishment phase evaluation Key findings

Results

PAGE 1

Barriers/challenges for establishing the Trials Solutions

Different funding and contracting systems across government agencies: Created difficulties in identifying eligible contracts to be moved to the Trials appropriation (e.g. some contracts were too complex to ‘unbundle’).

While the evaluation studies did not identify any solutions, parts of or all contracting templates and boundaries (e.g. service delivery regions) could be standardised across social sector agencies. This would help to identify funding that can be transferred to the Trials appropriation and potentially improve the ability to coordinate services across government in the future.

Complex decision making structures: Seeking high level decisions within and between government agencies was time consuming and resource intensive leading to delays in the establishment of the Trials.

Documentation indicates the establishment of the Joint Venture Board and Trials Director is expected to strengthen collective cross-agency ownership. This may reduce resource demands and speed up decision making for the Trials.

Other competing government initiatives: In some instances the Trials were seen as competing with other initiatives. For example, some regional service providers participating in High Trust Contracting initiatives were now facing dual lines of reporting.

CIs and NGOs reported meeting with relevant government agency stakeholders to coordinate how initiatives were being delivered in locations. Government agencies and providers involved with delivering other government initiatives in Trial locations should also actively seek to coordinate their activities with CIs and NGOs.

Changes in funding and contracting relationships: Some providers felt threatened by the change in contract management to CIs and NGOs. In some cases they were initially unwilling to engage with CIs/NGOs.

In some instances providers may face significant changes to their contracts and their feeling threatened may be unavoidable. CIs and NGOs reported managing relationships with affected providers was very important.

Scope of criteria used for contract selection: Fewer contracts were identified for inclusion within the Trials appropriation than anticipated. This may impact on the ability of CIs or NGOs to bring about change.

Key stakeholders suggested eligibility criteria for including contracts from the partner social sector agencies could be revised and additional contracts identified to increase resources available to the CIs and NGOs.

Keeping a low profile for the Trials during scoping: A need to establish the Trials without drawing widespread attention within communities may have affected the level of community stakeholder ownership and engagement in the Trials.

To improve engagement CIs and NGOs reported meeting with community stakeholders as early as possible in the planning process to introduce the Trials and seek input into planning processes.

Gov

ernm

ent

Tria

ls

Establishment period

Inputs Tasks OutputsImmediateoutcomes

Intermediateoutcomes

Ultimateoutcomes

2011 2012 2013

Collaboration between community organisations • Survey results for collaboration were similar for CI and NGO locations.• Before the Trials almost all organisations reported working with others (96%).• Quality of collaboration was good but there is room for improvement in areas such as coordination,

dependability, and goal alignment.

Engagement in the Trials• Awareness of the Trials was high (81%) among community stakeholder organisations and over half (54%)

had already met with CIs or NGOs about the Trials.

Enablers for establishing the TrialsGovernment level factors:• existing mechanisms within government that facilitated the transfer of funding across government agency

appropriations• other complementary government initiatives.

Community level factors:• a history of action around youth issues• existing community networks that helped to link CIs and NGOs with community stakeholders.

Trials level factors:• development of systematic and transparent processes to prioritise options for Trials locations• the autonomy and authority given to CIs and NGOs to carry out their tasks• management of contracting relationships at a community level was seen by some providers as an opportunity

to improve services.

Results are reported on: baseline levels and quality of collaboration between community organisations; stakeholder engagement in the Trials; and enablers, barriers or challenges faced by stakeholders during establishment.

Trialling New Approaches to Social Sector Change (the Trials) aims to change the way cross-agency and community resources are used to improve service delivery and outcomes for young people. Two models for organising service delivery are being tested:

Evaluating the TrialsThe evaluation is being led by the Centre for Social Research and Evaluation within the Ministry of Social Development in partnership with the Ministries of Justice, Education, Health and the New Zealand Police. The overarching evaluation aims to identify and assess:

Summaries of evaluation findings will be delivered at the conclusion of key phases of the Trials initiative.

• a Non-Government Organisation (NGO) model where an NGO is given authority via contractual arrangements to deliver the Trials.

• a Committed Individual (CI) model where a well regarded community figure is employed as a public servant and delegated authority to deliver the Trials.

• any similarities or differences between the CI and NGO models• any enablers, barriers or challenges encountered• whether the Trials will lead to better approaches to service delivery• whether the Trials will contribute to improved youth outcomes.

This A3 presents findings from the establishment period of the Trials. Future evaluation activities will focus on progress towards key outcomes.

PurposeThe purpose of the establishment phase evaluation was to understand what worked well, what did not and to provide evidence to enhance decision making by key stakeholders.

Sources of evaluation information• Individual and focus group interviews with the Trials Project Team, CIs,

NGOs.• Review of key documents.• Analysis of key themes from CI and NGO log books.• A survey of organisations in the six Trial locations conducted from late

May to early June 2011.

Many community organisations are already engagedWhile the Trials are still in their early stages, results indicated most community organisations had heard about them. Many organisations had also met with CIs or NGOs about the Trials.

Survey results indicated the proportion of organisations working together to deliver youth outcomes before the Trials started was very high. However, there are opportunities to improve the quality of how organisations work together, for example improving the coordination of services and programmes.

No major initial differences between CI and NGO modelsNo major differences were observed between how the CI and NGO models were being implemented or any initial impacts they were achieving. Some differences were found in the characteristics of the Trial locations that may influence how the Trials are implemented and the results they achieve.

Many barriers to implementation can be addressedGovernment agency level actions that could enhance the delivery of the Trials include standardising all or part of the contracting templates and boundaries within and between agencies, and streamlining reporting and decision making structures.

Trials specific actions include broadening the criteria for including contracts in the Trials, and providing ongoing opportunities for community stakeholders to have input into the Trials.

Immediate** Intermediate** Ultimate**Inputs*(Trials Project Team activities)

Tasks(CI and NGO activities)

Outputs(first six months)

Social sector governance

Status• Joint Venture Board and Trials

Director positions established

Enablers• Strong leadership from Trials

Director

Barriers/challenges• Complex decision making

processes were time consuming

Contracts and funding

Status• Relevant contracts identified and

moved to MYD appropriation

Enablers• Existing mechanisms enabled easy

transfer of appropriations

Barriers/challenges• Differences in contract templates

and boundaries across government agencies created challenges in identifying what was in scope

• Scope of contract selection criteria limited the number of contracts identified

Locations identified

Status• Six locations selected

Enablers• Use of criteria to aid transparent

location selection process

Barriers/challenges• Short time frame between agreeing

on the number of Trials locations and having to recruit CIs and NGOs

Recruiting CIs and NGOs

Status• All CIs and NGOs recruited. Some

NGOs experienced delays in recruiting project managers

Barriers/challenges• Having to initially keep a low profile

meant the Project team was unable to fully engage with communities. This was linked to some stakeholders feeling they did not have a fair opportunity to apply for CI or NGO roles

• Limited time to recruit, orientate and train CIs and NGOs

Develop Trials plans

Enablers• Other government initiatives that

provide opportunities for collaboration (e.g. CRM)

• National and local support to participate in plan development

Barriers/challenges• Competing government initiatives• Relatively short time frames to

engage stakeholders and develop draft plans

Manage provider contracts

Enablers• Providers willing to participate and

keen to improve performance

Barriers/challenges• Trials specific activities within some

regional contracts had to be moved to a separate contract leading to dual reporting lines for providers

• Contracts being held by providers who are not based in Trials locations (e.g. a Hamilton based provider holding a contract for services in Taumarunui)

• Balancing time spent developing Trials plans and managing contracts

Engage key stakeholders

Enablers• Existing community networks and

fora facilitating engagement• Ability to link directly with youth and

government agencies

Barriers/challenges• Some stakeholders unaware of the

Trials, the role of CIs or NGOs or how they were recruited – potentially leading to mistrust

Establish governance groups

Barriers/challenges• Ensuring membership is balanced• Ensuring a broad range of needs are

represented by board members

PROCESSES OUTCOMES

Social Sector Trials: Evaluation framework July 2011

Trials plans signed and actioned

Status• At the time of writing this report

Trials plans were works in progress and still in draft form

• Draft Trials plans included the four youth outcomes and references to improved collaboration

Contracts aligned to Trials plans

Status• Expected to be included as part of

the Trials plans

Stakeholder participation

Status• Forums, meetings and workshops

have been held with providers• Survey findings indicated most (81%)

stakeholder organisations were aware of the Trials and half (54%) have participated in Trials meetings

Governance groups

Status• Terms of Reference have been

developed by CIs and NGOs• Governance groups have been

established in each location• In some locations working groups

have also been established

• Almost all (96%) organisations reported they were collaborating with other organisations

• A third were meeting with other organisations at least weekly (29%), a third at least once a month (31%)

• Local schools, Police and CYF were the most common organisations worked with

• Quality of collaboration was generally good

• There may be scope to improve areas such as alignment of goals, frequency of joint planning, coordination, and dependability

Perceived needs:• Common perception among CIs

and NGOs that services were uncoordinated and organisations were not working together well

Improved service deliveryfor youth outcomes

To be explored in future evaluation activities

Improved action on the causes of truancy, offending, alcohol and other drug abuse and non-participation in education, training and employment

To be explored in future evaluation activities

• Reduced truancy• Reduced offending• Reduced alcohol and

other drug abuse• Improved participation

in education, training and employment

• Of the community organisations surveyed most were working towards improved participation in education, training or employment (77%)

• Over half were working towards reduced alcohol and other drug abuse (62%), youth offending (57%) and truancy (53%)

Engagement with youth outcomes

Baseline figures:

• Improved coordination of service delivery

• Sustainable family interventions • Refocusing services from treatment

to prevention• Workforce development• Better engagement of communities

in planning• More transparent use of resources• Less resource being used on

administration

Fit between services and youth outcomesPerceived needs:

PAGE 2

Collaboration between organisationsBaseline figures:

See high level baseline outcome indicator report

To be further explored in future evaluation activities

Evaluation Framework• The evaluation framework is based on key expectations and actions for the Trials.

These were identified in the document review and stakeholder interviews.• Findings on enablers and barriers have been mapped onto the inputs and tasks of

the framework.• Baseline survey data and relevant stakeholder interview findings have been

mapped onto the outcomes.• This framework will continue to be refined as new information becomes available.

Notes* Only those inputs that were identified during the course of the evaluation have

been identified and this report may not reflect all inputs.

** This report focused on the Trials establishment period (inputs and tasks). Only baseline information for outcomes has been provided.