evaluation of the australian men’s shed...

Evaluation of the Australian Men’s Shed Association

PART 4 EVALUATION OF THE NATIONAL SHED DEVELOPMENT PROGRAMME Final Report

31 AUGUST, 2016

Contents PART 4. NATIONAL SHED DEVELOPMENT PROGRAM 4

Executive Summary 4

Main Messages 7

Recommendations 8

EVALUATION QUESTIONS 9

Are AMSA’s business processes, including the administration of the National Shed Development Program (NSDP), effective and efficient? What is the experience of members with the NSDP and how could it be improved? 9

Evaluator observations 9

Summation of findings on effectiveness and efficiency 14

Summation of findings on the experience of members with the NSDP and how it could be improved 19

ATTACHMENT A: DATA SOURCES 21

Source 1: Business process review - NSDP 21

Source 2: AMSA staff and Board member interviews 27

Source 3: Departmental stakeholders group interview/focus group 29

Source 4: Member sheds and other potential applicants surveys 30

4 SIGGINS MILLER FINAL REPORT PART 4: NSDP EVALUATION REPORT AUGUST 2016

PART 4. NATIONAL SHED DEVELOPMENT PROGRAMME Executive Summary

The Program

Since 2010, AMSA has been funded by the Department of Health to administer the National Shed Development Programme (NSDP). The NSDP is funded annually with $800,000 (to be dispersed in 2 Rounds of grant funding to eligible Men’s Sheds per financial year, through a competitive application process); and with $200,000 annually to cover the administrative costs of the program. This $1M per annum funding constitutes the majority of funding to the AMSA, under its Funding Agreement with the Department of Health.

As part of a broader evaluation of AMSA, two questions were posed to evaluate the NSDP:

(1) Are AMSA’s business processes, including the administration of the National Shed Development Programme (NSDP), effective and efficient?

(2) What is the experience of members with the NSDP and how could it be improved?

The objectives of the NSDP (as stated in the Funding Agreements) have changed to reflect the stage of development of Men’s Sheds in Australia. That is, rapid growth in the initial years (when the NSDP objective was to support the establishment of new Men’s Sheds) followed by consolidation of the large number of Men’s Sheds now operating – with the NSDP objective now to support the sustainability of Men’s Sheds and also support men’s health and wellbeing activities.

To achieve these objectives, AMSA’s responsibilities have been to administer: the promotion of the NSDP grants; the application process; the receipt, collation and ranking of applications; the submission of recommended awardees to the Department; the official notification of outcomes to successful and unsuccessful applicants (after a separate notification of successful applicants managed by the Department); and the disbursment of funds. These are all within specific time frames, determined by the Department.

AMSA’s role

AMSA’s role in the NSDP is at the ‘execution’ end of the process:1

Policy formulation Interpretation Execution

Department AMSA

It is important to note that AMSA ‘administers’ and does not ‘manage’ the NSDP process. The Australiasian Council of Auditors General (ACAG) makes this distinction:

‘In order to achieve the specified objectives each participant will need to be given the means, namely the necessary authority and resources.’2 [italics added].

This distinction has proven important in assessing the data, attributing causality, assembling the contribution story and consequently in assessing AMSA’s performance. The Funding Agreement provides AMSA with the resources to administer the NSDP, but with little or no authority.

Approach

In arriving at the following summary and recommendations, the Evaluators have: interviewed key stakeholders, analysed survey results, examined NSDP related documentation and processes. Where

1

Australasian Council of Auditors General (ACAG). Effective Public Sector Accountability. http://www.acag.org.au/epsa.htm

2 ibid

http://www.acag.org.au/epsa.htm


assertions have been made about the administration of the NSDP program, we have sought additional information or clarification and checked documents and procedures to evaluate stakeholders’ perspectives against the available evidence.

Stakeholder views

The two key external stakeholder groups consulted in the evaluation (the Department staff, n=5 and the Men’s Sheds representatives who had applied to the NSDP, n=213) had quite different views on their satisfaction with AMSA’s administration of the NSDP, its effectiveness and efficiency. The Departmental views were on the negative end of the spectrum, while the majority of Men’s Sheds that had used the NSDP (successfully or unsuccessfully) were either satisfied or very satisfied with AMSA’s administration. Where Men’s Sheds were negative about the NSDP, it tended to be about apsects of the program where AMSA has no authority (e.g. the schedule, the elgibility criteria, the priority groups stipulated, the number and types of questions asked and the amount of funding available). These are issues in policy formulation and implementation that are within the Department’s authority. Where the feedback from Shed stakeholders was negative about AMSA, it was sometimes a case of ‘shooting the messenger’ since AMSA’s role is to execute the instructions of the funding body.

Where the Department was critical of AMSA these were often in the areas of application length and complexity (yet the information reqired in the application process reflects the stipulated requirements of the Department), in the reporting of the administrative costs of the NSDP (as separate from the disbursement of grant monies) and in regard to the overall quality of the program administration. Concerns were raised about different types of applications forms for different categories. The data presented do not support the need for this approach, but such administrative decisions would benefit from discussion about the overarching purpose and goals of the NSDP.

Internal AMSA stakeholders (Board members and staff) pointed to the resource intensity of the program, but felt the effort was warranted because of the benefit to Sheds. Board members (providing the State perspective) reported some disengagement of Sheds due to perceived prioritisation of specific groups ‘based on postcode’. Other issues raised were: the scheduling problems (with references to specific historical incidents causing delay outside AMSA), the tight time frames for application and decision making and the complexity of the whole process.

Emerging issues

From the Office Holder surveys, stakeholder interviews, the document review and the business process review it was evident that the two key issues that are impacting on effectiveness, efficiency and quality are outside the current authority of AMSA. First, over six years of operation, the program has been changed at least seven times, i.e. there has been little stability in the program, with frequent changes to requirements. Many of the changes have coincided with change of government or change of Department personnel, resulting in schedule delays or additional steps in the administration process.

Second, the schedule for the two annual rounds of NSDP funding (although improved in the most recent Round) provides a disproportionate amount of time for the less complex processes, reducing the time available (for both the Sheds and AMSA) to complete the more complex processes. There is little or no time allowed for internal review, evaluation and quality improvement. These are whole-of-program issues that are beyond the scope of this evaluation of AMSA’s effectiveness, yet AMSA’s performance cannot be assessed without acknowledgment of these key influences on its ability to perform effectively.3

3 According to Mayne (2008), a credible account of a program’s performance must address the question. “How much of the outcome is the result of the program, and how much is the result of other causes?” Mayne, J. (2008) Contribution Analysis: An approach to exploring cause and effect, ILAC methodological brief, available at http://www.cgiar-ilac.org/files/ILAC_Brief16_Contribution_Analysis_0.pdf

http://www.cgiar-ilac.org/files/ILAC_Brief16_Contribution_Analysis_0.pdf

http://www.cgiar-ilac.org/files/ILAC_Brief16_Contribution_Analysis_0.pdf


The NSDP part of the Funding Agreement arguably constitutes outsourcing of the administration of the grants. Yet the process is not fully outsourced – with every step of the process and every document being closely checked and requiring approval by the Department. Clear delegation of some authority to AMSA to make changes within agreed parameters would save the resources of both parties.

Recent developments

A more recent change at government level (in the approach to accountability and transparency for grant funding) has been the catalyst for a more collaborative approach between the Department and AMSA to consolidate or update procedural and governance documentation and to more clearly articulate the evaluation process (i.e. ranking and selection) to be used by AMSA.4

Another concurrent development has been the invesitigation of the feasibility of transferring the NSDP application process to an online system. This has the potential to significantly enhance the ease of application for those Sheds that have computer access and good internet connectivity. It also has the potential to shift resources of AMSA that are currently devoted to data entry and manual or paper based processes towards increased promotion of the NSDP, quality control and auditing of applications and greater ease of reporting to the Department. However, unless adequate time and resources are allocated to testing and piloting any new system, as well as the maintenance of the legacy system during a transition period, it risks repeating the program’s past history of frequent ad hoc or reactive change and therefore risks the currently good reputation of the NSDP with the Men’s Sheds.

AMSA’s performance of it NSDP responsibilities

With all the above caveats in mind, and acknowledging the recent developments descibed above, the Evaluators concluded that between 2010-2015, AMSA

met the original NSDP objective of supporting the establishment of Men’s Sheds (2010-2014);

is meeting the new objective of supporting the sustainability of Men’s Sheds;

is showing early signs of meeting the requirement to support men’s health and wellbeing, although it is too soon after introduction to assess;

has met all its timelines, in spite of using a largely manual system of data entry and supported only by spreadsheets (however this is not sustainable);

for the most part, has maintained quailty (in terms of accuracy, consistency, service to applicants, meeting deadlines, transparency to applicants). [Where accuracy issues arose they were one-off, due to extenuating or external circumstances linked to the contextual issues described above. Complaints by some stakeholders of poor quality were found to be issues of old technology and systems rather than of quality. Process level issues are likely to be partly addressed by: the planned interoperability between AMSA’s membership database and the proposed online grant application system; and the agreed new NSDP Evaluation Methodology & Guidelines (June 2016)];

provides a good service and satisfactory/very satisfactory administration to applicants;

needs to improve the style of application forms, but this would be reliant on Departmental agreement on program stability and time frames;

needs to increase the amount of information and frequency of communication to its members about the NSDP (based on survey feedback);

proficiently manages and reports the disbursement of funds, but not the reporting of administration costs of the program (the latter is an issue for the financial review);

4 Some of these documents already existed, but this approach has consolidated and clarified procedures into one Guidelines document.


reports the program outputs (application numbers, application spread, and funds distributed); and has taken the inititaive to collect data on program outcomes;

has used the available resources and infrastructure economically, although heavily reliant on goodwill, pro bono contributions and unpaid overtime; and

has delivered the required outputs on time and within budget.

Sustainability

Some changes to the staffing arrangements that support the NSDP administration will enhance the sustainability of the program. These include training to provide back up and the development of operational guidleines to coincide with the proposed transition to an online system.

Decisions about the sustainability of the grants program itself (that is, the NSDP’s utility in meeting the government’s policy objectives) would be enhanced by clarifying the overarching program objectives, using the outcomes data that has been collected by AMSA (rather than just the outputs data) and deciding appropriate program measures that will inform decision making.

Main Messages

There are whole-of-program issues that should be resolved before any additional operational level change is introduced in AMSA administration of the NSDP.

The two most significant quality issues are frequent change at the policy level (lack of stability) and short time frames. Other operational quality issues cascade from these factors.

Considering the above and considering its size, infrastructure and resources, AMSA has been doing a relatively good job in administering the NSDP.

The current arrangements within AMSA are not sustainable into the medium term without some change to operational procedures, resource allocation and staff training, especially considering the proposed changes to technology infrastructure.

The Men’s Sheds are satisfied or very satisfied with AMSA’s administration of the NSDP.

The stakeholders in Men’s Sheds often do not differentiate between the policy decisions and AMSA’s execution of them.

The Department is, in effect, outsourcing the delivery of a service, but with little authority delegated to AMSA as provider. This has an impact upon AMSA’s ability to perform effectively.

Under the current arrangements AMSA is empowered to make only minor changes to its processing that would not necessarily address the key concerns of the Men’s Sheds.


Recommendations

1. Hold discussions between the funder and the provider to clarify the overarching purpose of

the NSDP, the appropriate goals in terms of program outcomes and consequent measures of

success.

2. A collaborative approach to addressing whole-of-program issues to achieve stability in the

NSDP and a realistic time frame that will support quality service delivery.

3. A collaborative approach to developing an implementation plan for new online NSDP

application processing, while maintaining the legacy system over an agreed period to ensure

equity of access for Men’s Sheds.

4. Consider shifting the participation of senior departmental and government stakeholders

from the outputs phase (distribution of funds) to the outcomes phase (acquittal process).

5. Clear written delegations from the Department to AMSA to facilitate prompt minor process

change and service improvement without risk to the program.

6. Use the opportunity of implementing a new online system to develop operational guidelines

and to equip other AMSA staff to provide back up for the NSDP.

7. Use the anticipated benefits of the new online system to allocate more resources to NSDP

promotion and quality assurance.


EVALUATION QUESTIONS: Are AMSA’s business processes, including the administration of the National Shed Development Programme (NSDP), effective and efficient? What is the experience of members with the NSDP and how could it be improved?

Evaluator observations

Note: Evaluator observations consider the information from all sources, including the business process and document reviews and stakeholder interviews. The observations look for: confirmation of data from other sources; anomalies between data from different sources; and any other issues or additional questions that arose from the consultation process.

Complexity of the process

Most AMSA Board and staff stakeholders described the application process as complex. The areas of complexity appeared to be in the identification of priority groups and categories of grants and the subsequent implementation of these changes in the Guidelines, application and assessment procedures. Stakeholders described the follow-up processes (disbursement of funds and acquittal) as relatively straightforward. In other words, the complexity is in the areas of activity in the four columns to the left of the diagram below. Office Holders’ (grant applicants’) views on of the process related largely to the complexity of the application requirements and the short time frame for applying.

Figure 4.1. The NSDP Process

Timeliness The issue of timeliness was raised by several of AMSA Board and staff stakeholders, with particular reference to a long delay that occurred in Round 10 (2015) after recommendations had been forwarded by AMSA to the Department. Advice from AMSA is that this was resolved and both parties are aware of the repercussions and are working to avoid a recurrence. Based on the most recent (Round 12) schedule, the timelines for the process are presented below in Figure 4.2. These timelines were changed from the previous few rounds (to allocate more time for application evaluation) and demonstrate that, of a 24-week application-decision-notification process, 21% (5 weeks) of the schedule is allocated to AMSA and the Sheds for advertising and receiving applications; 44% (10.5 weeks) is allocated to AMSA for checking, collating, assessing and making recommendations; 31% (7.5 weeks) of the schedule is allocated to the Department for accepting


recommendations and for media announcements and separate notification by letter from MPs; and 4% (1 week) is allocated to formally notifying the successful and unsuccessful applicants. Immediately after this notification, the next call f or applications begins the cycle again.

Therefore, considering the NSDP Process (Figure 4.1. above) and the NSDP Schedule (Figure 4.2.) the more complex processes do not appear to be allocated adequate time. Further, there is little or no time for regular review and evaluation of the NSDP program overall. When asked to suggest NSDP process improvement, Office Holders most commonly suggested improvement to the time frame of the application process (52 of 67 respondents about the application process,78%).5

Member awareness of NSDP

Perceptions of ‘member awareness of the NSDP’ was not a specific question to AMSA Board members (because in this study membership awareness is gauged principally from office holder surveys). However, the Board informants expressed some contradictory views on member awareness of the NSDP in their interviews, with the opinion that ‘the NSDP is not utilised by Sheds because it isn’t widely known’ (n=1) contrasting with the opinion that number of applications is high (n=1). Other Board informants suggested that, although aware of the NSDP, some Sheds have not applied in the belief that the postcode of their shed (n=3) or not being a member of AMSA (n=1) would exclude them from consideration.

In the Shed Office Holders survey, 312 responded to the questions about the NSDP; 213 (68.27%) said that they had applied for NSDP funding and 99 (31.73%) had not applied. Of those who had not applied for funding, 21 respondents (7% of all respondents/20% of those not applying) said that the reason they had not applied was ‘I didn’t know about the grants’.6

Other reasons for not applying were: not enough information about the application process n=20 (6%/20%); not enough time n=6 (2%/6%); not enough resources n=2 (0.6%/2%).

Therefore, the majority of office holders (93%) were aware of the grants and two-thirds of the Sheds had applied for a grant at some time. Lack of awareness does not appear to be the reason for Sheds not applying.

5 The original time frame of 6 weeks was reduced to 5 weeks in R9. 6 71 of the 99 respondents who had not applied to the NSDP provided the reasons why they did not apply. The percentages

in this discussion are presented the percentage of all 312 respondents/ and the percentage of the 99 ‘non-appliers’. Further detail is provided in the Survey Results section later in this Report.


AMSA promotes the NSDP in these ways:

AMSA website under ‘Funding’ tab-accessible to all site users

AMSA website under ‘News and Events’ tab-accessible to all site users

Bulk email Newsletters sent to every member Shed on our database - normally twice per year

Personal email to those who have inquired about the NSDP when the next upcoming Round is announced.

The promotion of the NSDP to non-AMSA members is more challenging because, unless an organisation contacts AMSA, knows about AMSA and looks at their website or has previously applied for a grant, they are not known to AMSA and cannot be included on any notification list.

Respondents were asked to provide any other reasons for not applying. The reasons predominantly were (in order of frequency):

Reason for not applying to NSDP Evaluator comment

Funded elsewhere (n=14, 35.00%) A possible indicator of sustainability

Don’t have a shed yet (n=9, 22.50%) Sheds at different stages of development

Restrictive criteria (n=6, 15.00%) Confirms observations of some Board members

Application concerns (n=5, 12.50%) These were in 3 categories:

the application was too complicated and demanding

not enough time to obtain quotes for the application [related to scheduling issue above]

there would be a small likelihood of success [this may be linked to perceptions of eligibility criteria]

In the graphs below, we present grant application data. Several factors may have contributed to high or low application rates. As the consultation responses and reviewed documents indicate, these factors may include: perceptions of ineligibility to apply; success in obtaining funding from other sources; previous winners not re-applying; addition or deletion of grant categories; changes to the application processes; or sheds moving through the establishment phase to a consolidation phase, requiring fewer or different resources.

The evaluators note that the NSDP process and structure has changed at least seven times over six years and may have influenced the program application rate data in the Figures below, e.g. the one-off round of specific grants for Aboriginal and Torres Strait Islander sheds, in addition to the usual grants; prioritising Sheds that had not previously applied; the removal of a grant category7 and the addition of a new category of grant (Health Improvement) with initially low application rates (n=17 in Round 11). [The application rate for this new category of grants increased in Round 12 (n=24)].

An examination of the data from Rounds 1 - 12 shows that the rate of applications for grants (as a percentage of the number of AMSA sheds) has declined since Round 4 (April, 2012). There was a slight increase in 2014, when Rounds 7 and 8 were combined due to the narrow time frames available in that year.8 The most recent Round (12) showed an increase in the percentage of sheds applying (from 13% to 20% of overall shed membership, see Figure 4.3. below) and warrants further tracking. There are some caveats on the data, but these do not affect the trend data significantly.9

7 Category 3: Shed Activities and Programmes (e.g. used for equipment for promotion and fundraising activities)

8 Due to factors associated with a change of government that are reported below.

9 Sources of data: Spreadsheets (Rounds 1 - 12), and data reported in quarterly and six monthly reports to the Department.

Caveats on data are: The reported numbers for AMSA shed membership do not always coincide with the time of NSDP application, so in two instances these have been estimated, based on membership numbers before and after the NSDP


Figure 4.3. Percentage of sheds applying for grants

As the membership of AMSA has increased, the number of sheds applying for NSDP grants has decreased. This could be for any (or a combination) of the reasons listed above. For example, the drop in application rates after Round 5 may be attributable to the decrease in amounts available for individual grants and to a change in Departmental priorities;10 and the drop after the combined Round 7/8 process may have been influenced by delays in the schedule associated with Departmental approval and announcement processes due to a change of government after Round 6. The effect was to disrupt the NSDP schedule for one year.11

The peak in applications was in Round 4 (April 2012) where 458 Sheds applied and where there was also a one-off round of specific grants for Aboriginal and Torres Strait Islander sheds, which added 45 grant applications to the annual figure (503).

application period. The data used in the graphs do not differentiate between AMSA and non-AMSA applicant numbers. The evaluators do not consider these data issues to be significant because the purpose of the graphs is to demonstrate trends only. For example, in Round 11, only ten (8%) and in Round 12 only eight (4%) applications were from non- AMSA members. 10

Round 6: maximum grant reduced from $10,000 to $8,000; Guidelines changed to give priority to Sheds that had not previously received funding. 11

These delays occurred in the approval of Round 6 outcomes and the approval of documents and procedures for Round 7. This resulted in the need to combine Rounds 7 and 8.

0%

10%

20%

30%

40%

50%

60%

70%

R1 R2 R3 R4 R5 R6 R7,8 R9 R10 R11 R12

Percentage of Sheds applying (based on AMSA registered sheds)


Figure 4.4. Shed application rates

The contractual relationship

As reflected in Funding Agreements and the operational interactions between the Department and AMSA, the NSDP part of the funding agreement arguably constitutes outsourcing12 of the administration of the grants.

Yet the process is not fully outsourced – with every step of the process and every document being closely checked and requiring approval by the Department. The reason for this is unclear from the documents. It appears to be historical or could simply reflect the personalities involved. Some Departmental stakeholders pointed to ‘quality’ failures, but when investigated by the Evaluators these were rare and sometimes the result of the ‘too much change/too short a timeframe’ issue that has already been raised or the result of an issue in a higher office in the department – not at all attributable to AMSA.

Whatever the reason, this way of working does not appear to be satisfactory to either party. It is time consuming and causes unnecessary delay and frustration.

The entire NSDP is arguably a low risk grant, based on the Department of Finance’s Guidelines.13 From the surveys it appears that if there is any reputational risk, it lies with AMSA because its stakeholder group only interact with AMSA throughout the process and some stakeholders already think AMSA controls the process.

Clear, written delegation of some authority to AMSA to make changes within agreed parameters would save the resources of both parties and the proposed introduction of an online system and the consequent process changes involved may provide the opportunity to test such new delegations of authority.

12

A couple of definitions: Outsourcing is a practice used by different companies to reduce costs by transferring portions of work to outside suppliers rather than completing it internally. Outsourcing is an effective cost-saving strategy when used properly. www.investopedia.com/terms/o/outsourcing.asp Outsourcing is an arrangement in which one company provides services for another company that could also be or usually have been provided in-house. searchcio.techtarget.com/definition/outsourcing

13 Commonwealth Low-risk Grant Agreement Risk Tool (July 2014). https://www.finance.gov.au/sites/default/files/Risk-Tool-July2014.docx

0

100

200

300

400

500

600

700

800

900

1000

R1 R2 R3 R4 R5 R6 R7,8 R9 R10 R11 R12

Shed applicant rates

Shed applicants Shed membership


Sustainability

Some aspects of the NSDP may not be sustainable into the future. First, AMSA’s administration has heavily relied on the knowledge, skills, enthusiasm and institutional memory of the Program Manager. There is some back up available from other staff (e.g. the membership Manager who assists with the selection and ranking process by providing secretariat support, and who has a working knowledge of the NSDP). There is also a new part-time receptionist (in training), who has the potential to deal with initial inquiries.

The proposed transition to an online application will, in the short-medium term, require more resource input rather than less. In addition, the latest NSDP procedures require AMSA to provide detailed feedback to unsuccessful applicants on request, which is adding a new process at an already busy part of the NSDP cycle.

The role of the Program Manager may need to change to a less hands-on, more supervisory and quality assurance role – delegating some aspects to others in the team and training them accordingly, with detailed written guidelines to document the new processes.

Second, if numbers of applications do not increase (as a percentage of the overall number of Men’s Sheds), the utility of the program (in relation to the effort) may be called into question. Low application rates may be an indicator of Sheds achieving sustainability, or they may be an indicator of criteria that exclude the majority of Sheds. Whatever is the case, the government investment in the NSDP may decline in the future or cease altogether. All these considerations need to be informed by regular review and monitoring of outcomes (in addition to outputs). AMSA has maintained the collection of such data. This practice should continue and the data should be used. This is also where the absence of clearly articulated overall program objectives is a significant issue because clear program objectives would guide decisions about the use of outcomes data and the measures of success.

Summation of findings on effectiveness and efficiency

Definitions:

Effective – The extent to which the objectives were achieved. Takes into account timeliness and quality, management, credibility, how success is measured and monitored by AMSA.

Efficient – A measure of how economically resources/inputs (funds, expertise, time, materials etc.) are used to achieve objectives/results. For example, have resources been used economically? Are there alternatives for achieving the same results with fewer funds? Are deliverables achieved on time and within budget? Are activities carried out as simply as possible? Are decisions made as close to where services are delivered as possible? Are overheads as low as possible? Are duplication or conflicts addressed and resolved? Is innovation present?

The Department has provided its preferred definitions for the terms effective and efficient. The following summation of findings assesses the available data (drawn from document review, interviews with key stakeholders (AMSA staff and Board members; Departmental staff) and the Office Holders’ survey).

Are the NSDP business processes effective?

The extent to which the objectives were achieved. Note: In the publicly available documentation about the NSDP (in both AMSA produced materials and the Department of Health website14) there is no clear statement of the program’s objectives.15

14

http://www.health.gov.au/internet/main/publishing.nsf/Content/male-health-programs 15

For example, the information describes: What the NSDP does (provides direct financial assistance); who gets priority; how much is available and how often; information on eligibility and the range of definitions of what is a Men’s Shed.


That is, the descriptions of the NSDP do not articulate an overarching objective in terms of preventive health or outcomes other than distributing funding.

The objectives for the NSDP as outlined in the funding agreement has changed over the program’s lifespan. From 2010-June 2014, the stated objective16 of the NSDP was to support the establishment of new Men’s Sheds. In the last two funding agreements (since July 2014), the NSDP purpose is stated as ‘to support health and wellbeing activities and programs to support the sustainability of Sheds.’

Therefore, assessment of the effectiveness of the NSDP in achieving objectives should be considered in two phases.

Phase 1: supporting the establishment of new sheds

Phase 2: supporting health and wellbeing activities and supporting the sustainability of sheds.

Achieving Phase 1 objectives:

supporting the establishment of new sheds

In Rounds 1-3, data were not collected on a Shed’s status (i.e. new versus existing). During this period, based on the rate of the NSDP membership growth, the NSDP would appear to have been effective in supporting the establishment of new sheds, as evidenced by the overall rate of applications for funding matching the overall rate of growth of sheds nationally.

Data were collected from Round 4 onwards and are presented below in Figure 4.5. The data from Rounds 4-9 represent the rest of the period of the “Phase 1 objective.” These data indicate that the rate of new shed applicants decreased roughly in proportion with the rate of applicants from existing sheds and in proportion with the overall slowing of membership growth during the same period. Data from Rounds 10-12 indicate that, in line with the changed objectives of the program and the levelling off in membership numbers, new sheds have become a smaller proportion of NSDP applicants.

Figure 4.5.

One issue with the data is that, in requiring applicants to self-identify as either a new or existing shed, there is no definition of ‘new’ or ‘existing’ (e.g. <6 months or > x years) provided to them.

Further, the Funding Agreement reports over the Phase 1 period (R1-R9) did not stipulate reporting of outcomes in terms of funds awarded to new sheds versus existing sheds.

16 Stated in the Funding Agreements or variations

0

200

400

600

800

1000

R1 R2 R3 R4 R5 R6 R7,8 R9 R10 R11 R12

New versus Existing Sheds application rates

(no data available R1-4)

Total Shed applicants Total Shed membership

New Sheds Existing sheds


Conclusion: With the caveats of incomplete data and reliance on self-reports of status, it would appear that, over the specified period, the NSDP met the objective of supporting the establishment of new sheds.

Achieving Phase 2 objectives:

supporting health and wellbeing activities

The Department introduced a new category of grant (Health Improvement Activities) which commenced in Round 11. This new form of grant represents the Department’s implementation strategy to achieve the new objective of the NSDP, as stated in the Funding Agreement. The measures used to assess AMSA’s effectiveness in achieving this objective are (1) the application rate for grants in this new category; and (2) the distribution of funds for activities in this category. There have only been two completed application rounds for this new type of grant (R11, R12).

The initial application rate was 17 of 124 Sheds applying (14%); in the second round, applications increased to 24 of 180 Sheds applying (13%). Therefore, the number of Sheds applying for Health Improvement Activity funding has increased by 41%; but due to a larger number of Sheds applying in Round 12 for all categories, the percentage of Sheds applying for the health activity grants has remained stable over the first two introductory rounds.17 Over many rounds, the third grant category has historically attracted 15-30% of the grant applications (that is, the tools and building categories have historically been more popular with applicants). AMSA has been promoting the new Category 3 by supporting a lead shed to deliver a regional men’s health event.

The funds distributed for health improvement activity grants in R11 were $40,040 (with 17/17 applicants funded). In R12, the funds distributed were $43,235 (with a 19/24 applicants funded).

Conclusion: it is too soon after the introduction of the new category of funding to assess whether AMSA’s implementation of the new grants has been effective. Data from R13 will be important in determining whether the small upward trend in applications and funds distribution continues.

supporting the sustainability of sheds

The number of AMSA sheds grew rapidly between Rounds 1 and 4 and Shed application rates (Figures 4.4. and 4.5.) then levelled off in parallel with the levelling off of member Shed numbers. Office holder surveys found that 35% of Sheds who had not applied for AMSA funding had not done so because they are funded from elsewhere. Shed membership levels have been sustained at the 850-950 mark since Round 5, with a small, temporary drop in membership at Round 10 that coincided with the introduction of membership fees and is not attributable to the NSDP.

Conclusion: Data from “Phase 2” (Rounds 10-12) indicate that NSDP activity maintained its level and is showing a recent increase; and that the predominant beneficiaries of funds are now existing Sheds, suggesting that the changed focus towards sustainability has been achieved.

Take into account timeliness and quality, management, credibility, how success is measured and monitored by AMSA.

Timeliness

The time frames for AMSA to administer the program have fluctuated, but have tended to be short. Prior to Round 12, the 3 previous rounds allocated about thirteen weeks for advertising, handling inquiries, receiving, collating, ranking and reporting recommendations, and this is largely performed by one person (with a couple of volunteer panellists assisting at the evaluation stage).18 In Office Holder surveys, the most commonly identified area for improvement (n=52) was in NSDP time frames. The problems from applicants’ perspective were: the application timeframe is too short (e.g.

17 Another consideration is that funding for First Aid/CPR training was moved from Category Three to Category One in R13. Therefore, this health-related activity is more difficult to measure as it is included with non-health activities. 18

Round 9 was 13.5 weeks, Round 10 was 10 weeks, Round 11 was 13 weeks. Time frames tended to be longer earlier in the life span of the program (e.g. Rounds 5&6 allocated around 18 weeks for the same administrative processes).


to obtain quotes, prepare documentation); and the decision making process is too long. Evaluation panellists reported that the short time frame precluded extensive checking to validate the claims of applicants. Insufficient time appears to be allocated to the most complex parts of the process (both for applicants and for AMSA, see Figure 4.1.). Time frames are causing concern to stakeholders, but they are out of the control of AMSA.

Conclusion: It is remarkable that the aspects of the process that AMSA can control have been managed on time by AMSA, considering that AMSA is using a largely manual system of data entry and supported only by spreadsheets. However, this is not sustainable because it relies on long working hours (including weekends), mostly of one key person.

Quality

The current system involves three levels of risk due to human error or other factors. All these levels of risk can impact upon quality and some examples are provided here. First, at the applicant level: relying on applicants to provide accurate information (e.g. re their electorates and their membership); requiring applicants to estimate the extent of drug or alcohol or mental health or relationship issues among their membership is unreliable and prone to error or abuse.19 Second, at AMSA processing level: the manual data entry of 130-200 applications into multiple data fields is prone to error; as is the application of multiple scoring systems to rank applicants in different priority groups; and the checking of applicant bona fides /validity is also largely manual (e.g. referring to AMSA membership database) or relying on local knowledge of key individuals or checking the attachments to large hard copy applications. Third, at the policy level: changes to NSDP requirements and accompanying documents or processes have been frequent (at least 7 changes over 6 years); and the existing timeframe does not allow for considered review and evaluation to enhance quality.

In examining the impact of these risks at the second level (AMSA processing) there was not a great deal of evidence of poor quality resulting from the identified risks. There were some issues of transparency of process (e.g. in the evaluation process) but these have been addressed in discussions with the Department and the consequent internal guidelines that have been developed recently. These guidelines will also serve as a form of ‘Operations Manual’ to support consistency and sustainability of the administrative processes.20 Many of the issues that the Evaluators identified were related to old technology and systems rather than poor quality. For instance, we found very little inaccuracy, little evidence of missed deadlines or of poor quality service; and little evidence of not having met the performance indicators.

Quality issues raised by stakeholders were to do with: (1) reporting and transparency (largely related to application evaluation/ranking process);21 (2) the unwieldy nature of the spreadsheets; (3) errors, for example in electorate data;22 and (4) the unwieldy, complex nature of application forms.

Conclusion: There are risks to the program that can affect quality. These are whole-of-program issues. The two most significant quality issues are frequent change (lack of stability) and short time frames. Other quality issues cascade from these factors. Of the issues that are within the control of AMSA, the process level quality issues are likely to be partly addressed by the planned interoperability between AMSA membership database and website (reducing manual data entry and facilitating checking of some applicant claims); the agreed new procedures and documentation that were negotiated in December 2015; and the proposed shift to an online application process (which

19

We note that the Department requires that subjective data from applicants is secondary to other data, based on more objective sources. This raises the question of why the subjective data is collected at all. 20

The Guidelines are covered in more detail in the Document Review. 21

The financial reporting of the AMSA administrative process (as opposed to the financial acquittal of grant monies) is covered in the Financial Report. 22

Which were data entered by applicants during a period of electoral boundary change. An audit undertaken by the AMSA using the AEC search facility found complexities when metro postcodes may produce multiple electorate options and it was not uncommon for applicants to provide in their State electorate.


should eventually – but not immediately – free up AMSA time and resources to devote to quality assurance).23

Management

In the Office Holders survey, respondents expressed satisfaction with the overall management of NSDP processes. Where suggestions for improvement were made, they often related to whole-of-program issues (e.g. timeframes, eligibility criteria, the amount of funding). These issues are not within AMSA’s authority to change. AMSA-specific areas of operation that could improve were identified as (1) the complexity and size of the application forms and (2) communication (i.e. more information about the NSDP, more frequently communicated).

The grant funding disbursement and acquittal process was scored highly by AMSA stakeholders. The concern of Departmental stakeholders was largely with transparency of NSDP administrative funds, rather than grant monies.24

The application process is covered in more detail in Attachment A. (Business Process Review) below.

Conclusion: Areas within the ambit of AMSA to improve are the application process and communication with the Sheds about the NSDP. In terms of the application process, without some agreement with the Department, little except the format (better design), the means of accessing and submitting application forms (e.g. online) and the duplication of forms per category could be altered. After recent negotiations and checking of the compliance of forms by both parties (December 2015), all the questions on the form appear to be required by the Department. Issues of the length and complexity of the application forms appear to largely reflect the data required by the Department. AMSA has developed a revised application form to address these concerns. It is currently in draft form.

Communications could be enhanced with more stability and certainty about the NSDP content and with more lead time to promote and inform prior to subsequent Rounds.

Credibility

AMSA’s administration of the NSDP has credibility with AMSA membership, as demonstrated by overall satisfaction ratings from the Office Holders’ survey, the majority of respondents were either satisfied or very satisfied with the NSDP processes.

AMSA has less credibility with Departmental stakeholders, who expressed concern about the administrative processes.

How success is measured and monitored

There is a detailed discussion of the measurement of outcomes in the Business Process Review below (Attachment A). The Funding Agreements have included quite specific measures which are process-based rather than outcomes-based. AMSA has collected additional, outcomes-based data over several years.

Are the NSDP business processes efficient? Have resources been used economically?

Considering the infrastructure available to AMSA between 2010 and 2015, the available resources have been used economically, although heavily reliant on goodwill and unpaid overtime.

Are there alternatives for achieving the same results with fewer funds?

With further funding certainty from 2016-2019, AMSA is seeking to invest in enhanced IT support systems that would improve the administrative processes that support the NSDP. This may not necessarily free up funds immediately, however, because of the need to maintain a legacy system, at least through a transition period, for Sheds without computer access in order to ensure equity. In

23

See the IT Systems Review 24 See Financial Report


the medium term, the online alternative would improve quality and efficiency, rather than achieve significant savings. Further, the most recent NSDP Guidelines and the funding agreement add elements to the process (e.g. individual post-announcement feedback to applicants; delegation of responsibility to AMSA for checking electorate data) that will require additional resources. Therefore, the alternative involves better use of resources, rather than fewer resources.

Are deliverables achieved on time and within budget?

Yes, AMSA deliverables that are within their control are achieved on time.

The deliverables are also achieved within budget in the sense that the overall AMSA operations have been consistently within budget. The Evaluators note that until recently, AMSA-related costs have not been separately reported to the Department.

Are activities carried out as simply as possible?

No, because there are so many factors contributing to complexity. Some of these are internal, some are external to AMSA. However, for AMSA-administered aspects, key priorities could be to simplify both the application process and the evaluation (ranking) processes where it is empowered to do so. A shift to more computerisation through interoperable AMSA systems and online NSDP application would greatly enhance the simplicity of the administration.

Are decisions made as close to where services are delivered as possible?

AMSA has very little decision making authority in the NSDP. Its role is to implement decisions of the Department of Health. It can recommend only.

Are overheads as low as possible?

Yes, AMSA benefits from heavily discounted accommodation, IT and other services. Staff salaries are in the higher range for the community sector, but staffing numbers for the NSDP have historically been very small.

Are duplication or conflicts addressed and resolved?

There is evidence of cooperative, collaborative relationships between AMSA and the Department in regard to the NSDP.

Is innovation present?

Innovation is planned, as evidenced by changes in train to improve the IT infrastructure and software that will support new processes for the NSDP (and other AMSA business processes). The back to back cycle of NSDP Rounds has precluded internal review and evaluation; the NSDP requirements have changed regularly since its inception; the ongoing funding for the NSDP has been uncertain at different points; all changes – even minor ones – require departmental approval. None of these factors are conducive to innovation.

Summation of findings on the experience of members with the NSDP and how it could be improved

The respondents to these NSDP survey questions include both AMSA and non-AMSA members (Shed Office Holders). This is because the grants are open to all Sheds that meet certain criteria.25 Two thirds of the Sheds surveyed had applied for NSDP funding at some stage. If they hadn’t applied for funding, it was most likely because the Shed didn’t know about the grants, didn’t have enough information about the grants or didn’t need the funding (i.e. was funded from another source).

Overall, respondents who had applied for the grants were either satisfied or very satisfied with the process.

25 Non-AMSA shed members were not included in other data (elsewhere in the evaluation) about AMSA services for which they are not eligible.


It is understandable that respondents are sometimes unclear as to the role of AMSA in the NSDP (i.e. as the administrator and not the funder). In making comments, respondents often raised issues that are policy issues - where AMSA implements rather than decides.

Key areas of improvement suggested by the respondents were in the following areas, but not all are within AMSA’s authority to change.

Area to improve AMSA authority

Department authority

Application process – time allowed to apply

Application process – time to decide & notify

Application process – complexity (forms)

Application process – complexity (questions)

Application process – multiple forms

Communication – frequency and content

Communication - methods

Funding available – amount

Funding available – type

Priority groups – ‘restricted eligibility’

Feedback to applicants

Note: Attachment A (data sources) also contains additional detailed Evaluator Observations about the NSDP processes and documentation.


ATTACHMENT A: DATA SOURCES Source 1: Business process review - NSDP

Does AMSA record processes that guide the funding grants (NSDP) allocation process, selection of grant awardees and measuring of grant outcomes? The evaluators note that, as recently as December 2015, a discussion paper26 about the NSDP grants evaluation process (i.e. the assessment of applicant eligibility, ranking of eligible applicants and the allocation of grant amounts to successful applicants) was prepared by the Department of Health.

This followed the signing of a deed of variation to the agreement between AMSA and the Department on 25th June 2015. The variation included re-designed grant requirements.27 Changes included specifying socially disadvantaged groups in the contract28 such as: defence veterans, males suffering alcohol or drug abuse and males who need support due to relationship issues.’ A significant change to the process was the introduction of new selection criteria, with applications to be scored on how well they demonstrate: the eligibility of their proposed activity and how that activity meets unmet need at their shed; the value for money of the proposed activity; the shed’s organisational capacity to manage the project and funds; and how well their organisation manages risk.

The discussion paper also included detailed instructions for AMSA evaluation processes and scoring procedures. AMSA responded to the discussion paper in December 201529 with detailed descriptions of their internal processes, which had been developed iteratively and in consultation with the Department. AMSA raised some additional issues or opportunities for improving the process. Some of these additional suggestions were taken up. The Department made four recommendations in the 2015 paper:

1. “Application forms and NSDP Guidelines should match the selection process so they are not misleading or collecting information that is not used but not changed so much as to cause AMSA undue stress.

2. SEIFA data should not be provided to AMSA in data cubes but as a simplified spreadsheet that provides postal area code and a single score out of 100, the Department should keep a more detailed spreadsheet showing how this score was arrived at.

3. The entire process needs to be transparent and clearly defined. 4. Round 12 should be used a test case and feedback requested from the panel on ease of use.”

Round 12 is only recently completed and the time frame for this project precludes any meaningful assessment of the effectiveness of the process changes recommended by the Department for this round of grants. It is noted that the issue of measuring/evaluating grant outcomes was not covered in the Department’s discussion paper or recommendations. This issue is covered below.

Current process documentation

Internal AMSA documentation

Arising from the discussion paper in December 2015, a new document was developed and finalised in June 2016 called National Shed Development Programme NSDP Evaluation Methodology & Guidelines.

26 Department of Health (undated 2015). National Shed Development Programme – Grant Evaluation Process 27

The changes over recent contract renewals have been: the re-ordering of priorities, the move from funding to ‘develop’ sheds to funding to ‘sustain’ sheds (2014), then the removal of Indigenous-specific activity requirements (2016) the requirement to maintain an appropriate IT database (2016) and the requirement for AMSA to verify the electorate of successful applicants (2016). 28

The original Funding Agreement (May 2010) also specified these priority groups in Schedule A, Stage 1, p.28.; as did the Deed of Variation in 2012. Therefore, the change was the addition of three new priority groups. 29

AMSA (December 2015). Response – DOH Review, December 2015.


Note: the above document was produced after this AMSA Evaluation project had commenced (9 May, 2016) and it addresses many of the business process issues/questions that the reviewers had identified in the early stages of our research.

The Evaluation Methodology & Guidelines largely reflects the evaluation process proposed in the discussion paper and describes it in detail, step by step. It also describes:

Roles and Responsibilities of the Evaluation Panel

Conflict of Interest (declaration of and procedures in the event of)

Roles and Composition of Panel

Confidentiality requirements

Departmental documentation requirements (from AMSA)

Notification procedures (Departmental, then electorate and then AMSA)

Procedures for successful applicants (funds distribution and acquittal)

Procedures for unsuccessful applicants (debriefing)

Security requirements (documents and panel discussion)

The document also contains templates, some of which were already developed and in use by AMSA* but have been updated and consolidated into one guiding document –

Deed of Confidentiality

Compliance Checklist

Individual Score Sheets

Letter of Offer to Successful Applicants*

Letter to Unsuccessful Applicants*

Evaluation Progress Report (Acquittal)*

NSDP Round Evaluation Report

Internal documentation of the process also consists of spreadsheets (all applicants, all data); shortlisted applicants’ database evaluation scores; key information on successful applicants; and successful applicants’ financial record (tracking agreement signing, invoice receipt, internal payment processing and contacts with awardees).

Summary NSDP Round Evaluation Reports will accompany the spreadsheet-generated lists of recommended applicants to the Department. These reports will consist of: (1) an overview of any changes that have taken place in a particular round. e.g. changes in Guidelines such as modified funding limits in a category, eligible items within a category (2) Funding Round Timetable (3) Application Statistical Data

Process documentation for applicants

Process documentation for lodging an application is described below under ‘the application process’. The evaluation process is described to applicants in the Programme Guidelines for each Round. The process is presented in a simplified way which does not completely reflect the internal processes as described in the Evaluation Methodology & Guidelines. For example, it states that ‘In the event of a large number of applications within a specific category, applicants may be shortlisted based on the ABS socio economic and demographic data score issued’, whereas such shortlisting and other scores (ABS/Remoteness/Priority Groups sub-total + selection criteria30 sub-total) are the primary focus for the evaluation panel, as stated in the new internal guidelines (see comments on transparency below).

The process documentation for successful applicants consists of (1) a letter of offer that replaces a former funding agreement31 and stipulates the requirements for an invoice to AMSA for the award

30 Selection criteria = eligibility, need, value, organisational capacity, risk 31

Based on the Department’s view that contract was not required for small grants.


amount; (2) a follow up acquittal report template for awardees to complete and return to AMSA. This acquittal report was originally developed by AMSA after Round 2 of the NSDP and has been in use consistently since then, although not required by the Department.

The new Evaluation Methodology & Guidelines now include this process formally, and have allocated a time frame of ‘approximately six (6) months from completion of funding distribution.’ However, the data from these acquittal reports are not included in the documentation to be provided to the Department.

Final process documentation for unsuccessful applicants has consisted of a letter notifying that their application was unsuccessful and encouraging them to re-apply in the next round.

The new methodology reverses a previous policy that precluded individual feedback to unsuccessful applicants.32 As of Round 13, ‘individual applicants may nonetheless contact AMSA for a debriefing. Debriefing assists applicants to further improve their service as a Shed, be more competitive and potentially helping to achieve value for money in the future. It will not reveal information about other application proposals (e.g. details of pricing). Debriefing of both successful and unsuccessful applicants can include: reasons for the success or otherwise of the application; strengths or weaknesses and/or non-compliances; ways of improving future applications; an applicant’s compliance with the selection criteria.’

The debriefing is a new process that will occur at the same time that the call for applications is taking place. The Guidelines state that ‘This information will also be used to update NSDP documentation such as the FAQ and Writing an Application Guide.’

Evaluator observations (process documentation):

This review coincides/overlaps with recent discussion and change to the NSDP evaluation (ranking) process. Therefore, process documentation has recently changed and processes for allocating scores, ranking and reporting to the Department have changed while we were in the process of reviewing the documents.

These changes have addressed many of the issues identified by the reviewers in their preliminary examination of the NSDP business processes.

However, the addition of new post-award processes (debriefing of applicants) further congests an already busy inquiry period where currently only one staff member will have the detailed information and experience to debrief applicants. This suggests the need for training and equipping other staff members to provide debriefing and/or to handle detailed inquiries about the new application Round.

From observations during the site visit (and subsequent direct contact with the Shed membership) a significant cohort of shedders initiates telephone contact in preference to email inquiry or searching the website. Telephone inquiries are typically longer in duration than standard business calls. Observed telephone calls during the site visit included assistance for inexperienced computer users to locate documents on the website and clarification about information already received. A debrief as described in the Evaluation Methodology & Guidelines would involve identifying the application in a spreadsheet and (possibly) retrieving the original application to provide meaningful feedback on its strengths/weaknesses. Further, the expectation that the debrief process will inform updates of future documentation suggests that an additional process of recording/tracking of issues will be required. This is because of the timing of the call for applications and the preparation of materials. Issues identified in, say, a Round 13 debriefing, will

32

FAQs Round 12. ‘Question 14. Will individual feedback be provided to applicants? No. Unfortunately, individual feedback cannot be provided due to the widespread interest in the Programme and the large volume of applications that are received.’


not be addressed or able to influence changes and improvement until Round 15.

Selection of awardees - transparency

As stated above, the internal process of selection of awardees was closely examined and reviewed by the Department in consultation with AMSA in late 2015 – early 2016. The implementation of these new processes is recent and not assessable at the time of writing this report. The goal of the Department is that ‘the entire process needs to be transparent and clearly defined.’33 The Department also recommends that ‘Application forms and NSDP Guidelines should match the selection process so they are not misleading or collecting information that is not used.’34

Transparency has hitherto been addressed by (1) description of the evaluation/selection criteria and process in Programme Guidelines for applicants; (2) presentation of the scoring of the Evaluator Panel to the Department (in the Shortlisted applicants’ evaluation scores spreadsheet) as part of the recommendation process and (3) publication of the list of successful applicants to the broader membership and the public.

Evaluator observations (transparency):

The evaluation/selection process is described to applicants in a simplified form that risks misrepresenting the actual process, in terms of the emphasis on priority groups and how they are measured/scored.

The notification process is explained to the applicants in a simplified form that risks misrepresenting the actual process, e.g. the involvement of Members of Parliament in the notification steps is not transparent.

Questions appear in the application forms that are not explained in any of the accompanying documentation. These are:

(Part B) Federal Electorate in which Men’s Shed is located: [applicant provides the info] See Electorate Finder: http:apps.aec.gov.au/esearch/

Question 11 (new to R 13). Will your Men’s Shed Project generate local media interest?

Yes/No

These questions, therefore, do not meet the Department’s objective of transparency and of

matching what appears in application forms with the selection process.

Measuring Grant Outcomes

The focus of reporting and measuring NSDP outcomes, based on performance indicators in contracts and quarterly or half yearly reports by AMSA hitherto has been:

the number of applications

the number and spread of successful applicants, overall and per category

timeliness of the distribution of funds

delivery of two rounds of NSDP grants per financial year.

The recently developed Evaluation Methodology & Guidelines have introduced reporting templates to better account for the evaluation/selection process. However, the measurement of outcomes of the $800,000 per annum funding (as opposed to measuring the prompt and accurate distribution of

33 Department of Health (undated 2015). Recommendation 3. 34

ibid. Recommendation 1.


the funding) has not been a required part of the process. Internally, however, AMSA have been tracking and collecting data about outcomes using the NSDP Evaluation Acquittal Report since Round 3.

Similarly, a regular, consultative, evidence-based review of the process (by all key informants such as panellists, AMSA staff, Departmental staff and shed stakeholders) does not appear to have occurred over the 5-6 years of the NSDP operation. Review seems to have occurred on an ad hoc, reactive basis involving individuals in AMSA and the Department.

This absence of consultative, evidence-based review is largely due to the time frames that require a new Round to commence before the previous Round has been completed and reviewed.

Evaluator observations (measuring outcomes):

Qualitative data on the outcomes of NSDP funding have been collected at AMSA’s initiative since 2011-12 but the intensive schedule associated with grant rounds and the size of AMSA staff preclude its analysis and use as a quality measure.

Similarly, the tight schedule has precluded meaningful assessment of outcomes and resulted in frequent ad hoc, reactive changes that have affected the quality of processes and their supporting documentation.

The application process

The current (Rounds 12 just completed; Round 13 recently opened) application process consists of three types of application form (one per category) available in Word or pdf format and downloadable from AMSA website. Each application form is 12-15 pages long and, apart from one question, consists of essentially the same content. The variable question is ‘Question 11,35 Eligibility’ (of the item/activity for which funding is sought). “What activities/items are requested and for what purpose will they be used? Examples: [different examples are provided for each category].”

Applicants wishing to apply for funding in more than one category are required to complete more than one application form. Applicants are required to submit the original (signed) application; four (4) copies of the application; one (1) copy of the attachments. Attachments would commonly be 2 quotations, an Insurance Certificate of Currency (mandatory), relevant MOUs, building permits or permissions/endorsements.

Programme Guidelines Round 12 (15 pages) bring the recent changes to the attention of applicants and explain the NSDP in more detail; Preparing Your Application Handy Hints (4 pages) are also available to applicants, as well as FAQs (6 pages).

Application-related documents are proof read and checked by a Departmental staff member before release. The forms appear to be revised frequently (unrelated to contractual changes). For instance, Round 13 forms (available on the website from 10 June 2016) have changed (re-ordered questions, additional question).

Evaluator observations (application process):

Considerable time and care goes into the preparation of new application materials for each Round. Recent program changes have required additional revisions. It is apparent that many of the highlighted areas of application documents (instructions, mandatory requirements) have ‘built up’ over successive rounds of funding - anticipating likely errors or omissions, with the aim of averting incomplete or non-compliant applications and therefore minimising wasted effort by

35 Re-ordered as question 6 in Round 13 Application forms.


both applicants and evaluators. AMSA’s awareness of the varying capacity of the applicant groups is evident in the documents. These documents seem long and repetitive of the same information. The length of the documents is partly due to the relatively complex requirements that now apply. However, the repetition of information in different formats may be warranted considering that some Sheds continue to submit non-compliant applications (e.g. with no building permit for a building proposal), in spite of caveats and highlighted instructions embedded in the application forms and repeated in Guidelines, Handy Hints and FAQs.

The need for three separate application forms, however, is questionable. This is because only one part of the form is substantially different (one question, half a page in length). The data from Round 11 indicate that only 27 applicants (22%) applied for multiple grant categories (9 were successful in more than one category). In Round 12, 34 applicants (19%) applied for multiple categories and 4 of these applicants were successful in more than one category.

These applicants may be catered for without the need for duplicating full applications, such as a supplementary page(s) for the additional categories (if the current paper-based system continues).

Following recent discussions (2015-16) AMSA has produced a consolidated application form in draft form.

How could NSDP processes most effectively be transferred to a computer-based system? The IT Systems Review deals with the detail of how the NSDP process could technically be transferred to a computer-based system. Quotations have already been received for such an improvement, that achieve interoperability with a new online Membership database. Eventually, such a development would (in brief):

transfer the data entry to the applicant, changing the role of AMSA to checking and quality control.

store details of applicants who have previously applied and require only confirmation or updates by applicants

provide immediate acknowledgment and receipts to applicants

create a contact list of applicants for each round

reduce costs and resources for applicants, especially by eliminating the need for multiple hard copies of applications

immediately eliminate incomplete or invalid applications

enable automatic scoring and preliminary ranking of applications, using the data and procedures required by the Department

enable online access to panellists for additional qualitative ranking or decision making

generate reports and recommendations to the Department in the required format

store data that could be used for quality assurance, performance measurement, tracking trends and identifying program issues.

However, a period of transition would be necessary to avoid disenfranchising Shedders who are not computer literate or who do not have computer access or adequate internet connectivity. A transition period would involve (not an exhaustive list):

developing an agreed36 implementation plan for the project, especially adequate time to pilot and test

maintaining the legacy systems for an agreed period

agreed measures of successful implementation

36 Agreed between the Department and AMSA


a marketing campaign to Sheds

additional resources to support the implementation with telephone, email and online assistance to new users

data collection to track and evaluate the implementation, including a user feedback process; and

flexibility in the usual cycle of funding Rounds to allow for the implementation process.

Source 2: AMSA staff and Board member interviews

This group consists of ten informants closely involved in the administration and governance of AMSA (six Board members and four staff members). Not every informant answered in detail about the NSDP, due to varying levels of involvement in the process. Informants were interviewed separately either by telephone in person, or submitted an individual written response to the interview protocol.

Usefulness to the members versus resources required to deliver These informants were asked their views on AMSA’s management of the National Shed Development Programme (NSDP) in terms of the usefulness of the NSDP to members.

Six AMSA Board members were interviewed in May/June 2016. Five informants viewed the NSDP as highly useful to members and one informant viewed the NSDP as moderately useful to members.

All informants viewed the management of the NSDP as highly resource intensive.

Four AMSA staff members were interviewed in May/June 2016. Three informants viewed the NSDP as highly useful to members and one informant viewed the NSDP as moderately useful to members. Three informants viewed the management of the NSDP as highly resource intensive; and one informant viewed it as moderately resource intensive.

Conclusion: The consensus of the informants was that the NSDP was highly resource intensive and highly useful to members.

In your AMSA role(s) what has been your experience with the NSDP? What feedback have you had from members about the NSDP processes? Informants’ responses were in the following categories or themes:

The NSDP process

A complex, changing process

Informants described the NSDP process as ‘massive’ or ‘complex’ or requiring too much paperwork (n=3); with time frames that are too tight (for application and for internal evaluation and quality control) (n=2); and frequent changes to requirements that are confusing and frustrating to the membership (n=3).

Frequent changes (required to implement changes in Departmental priorities) had resulted in increased workload for AMSA, including increased levels of applicant inquiry and requests for assistance in understanding new questions or categories (n=4).

The evaluation of applications

Informants most commonly raised the issue of the validity of using the socio-economic eligibility measures based on the suburb/postcode of a particular shed (n=7). The types of concerns were: postcode is a poor reflection of the needs of a shed (n=2); it has caused resentment among the membership, including perceptions of discrimination or bias (n=3); it has led to applications of dubious validity or relevance because some sheds feel guaranteed of funding (n=2); it has caused additional workload for AMSA staff and Board members in explaining/justifying the policy.

Informants felt there is sometimes insufficient time to adequately verify claims made in some applications and that the evaluators rely heavily on local knowledge of Sheds and the knowledge of AMSA staff and the database (n=2).


Other factors were felt to create disadvantage, for example: sheds that operate under the auspice of another body have more facilities and resources to produce grant applications and the differences are very marked (n=1); and sheds that have received funding before are penalised for previous success (n=1).

Announcing funding, providing funding and acquittal of funding

Informants (n=5) reported that the length of time taken by the Department to announce the successful applicants was excessive and had caused concern to applicants and increased workload for AMSA.

Two informants felt that the amount of funding does not meet demand (based on the number of sheds nationally and the amounts available in each category of grant).

Two informants commented on the process of acquittal and evaluation of outcomes: that the effectiveness of such grant programs is difficult to measure; and that the processes required of the NSDP were not ‘particularly onerous.’

Perceptions of the NSDP

In the course of their interviews, informants provided the following perceptions of the NSDP:

Among the membership, there are historical and/or ongoing perceptions of geographic bias and/or cultural bias in the eligibility and selection (n=3)

It is a marketing tool for politicians (n=3)

Sheds do not fully understand the role of AMSA in the NSDP process (i.e. the administrator rather than the funder) (n=2)

It is virtually impossible to get a grant if you are not a member of AMSA and it is almost like a marketing tool for AMSA (n=1)

We note that there are plans to computerise the NSDP processes, but are there other ways the NSDP could be improved, and if so how?

Computerisation of the NSDP process

Six informants gave their views about the computerisation of the NSDP process. These were in two categories:

(1) Agreement with or neutral about the need to computerise, but with caveats (n=5). The reasons for agreement were: the need to reduce the workload for the AMSA manager (n=1); the potential to broaden the evaluation panel by having online assessment of applications by more people and therefore addressing perceptions of geographic bias (n=1). The caveats were: the membership will object or complain because they are computer illiterate or do not have access (n=4); the membership will need a lot more assistance in using any new computer-based system (n=1); once the new system is established, it will need to be maintained and regularly upgraded (n=1); the old system will need to be maintained in parallel so that sheds with access issues are not disadvantaged (n=1); the evaluation process can be enhanced by an online process, but the face to face discussion around a table will still be needed for short listing (n=1).

(2) The remaining informant declined to give a personal view and was adamant that any consideration of changing the NSDP system should be left to the new AMSA Board to consider and decide.

Other ways the NSDP could be improved

One informant suggested that ‘there is a need to develop more strategic questions that could draw out the reasons for the need for funds. The questions are however set by the funding authority and


in many cases sheds don’t understand them or the manner in which the questions are asked lead to a superficial response.’

One informant suggested that [the NSDP] ‘needs a complete re write, and a proper marketing and branding campaign to go with it, so that it becomes a distinctive and recognisable product in the market place (the sheds); [so] that the sheds understand what it is, where it comes from, and what use it is.’

Another informant suggested that the group of assessors (the evaluation panel) could be broadened and (with more time allowed in the schedule for others to do more preliminary research and validation of claims) enabled to spend more time in detailed discussion of shortlisted candidates.

Source 3: Departmental stakeholders group interview/focus group

Five departmental stakeholders (3 from the Preventive Policy Section and 2 from the Health & Sport Grants Section) provided the following feedback about the NSDP. This feedback was predominantly in the area of financial management and reporting and business processes.

What have been your experiences in regard to AMSA’s financial management (of the NSDP)?

Respondents felt that, from AMSA financial reports, it is difficult to determine how much is spent on administering the NSDP, as opposed to determining how much has been awarded in grants. This is considered to be an area AMSA would benefit from strengthening.

What are the Department’s expectations?

In financial reports, information on the grant monies is provided, but the staffing and administration costs are rolled in with core business activity costs. In future, the Department expects financial reports that separate (1) NSDP grant monies and (2) NSDP staffing and administrative costs. [The third category of AMSA core business income and expenditure is covered in more detail in the business processes and financial report sections of this review].

How well do the (NSDP) business processes work?

Respondents felt the NSDP processes are out of date; the form is badly designed; it is very long; it is very complicated and so it is off-putting to low literacy applicants, ‘and those are the sheds we most want to help.’ There are 3 forms to fill in and multiple copies to provide.37 Paper applications are entered into a large spreadsheet. ‘The process needs some sort of upgrading, and some professional form design.’

Some respondents commented that the Departmental staff spend considerable amount of time correcting what is submitted by AMSA. On exploring this theme, it emerged that the context where information required correction was in checking electorate data in AMSA reports of recommended successful NSDP applicants. (This responsibility had subsequently been delegated to AMSA under the new funding agreement). The provision of correct electorate data was seen as an important quality measure in the business process.

How could the administration of the NSDP be improved?

In addition to improving the financial reporting (as described above), respondents suggested that the process needs to be more transparent, needs to be simplified for applicants and needs to align with the government’s digitalisation agenda.

37

In internal Stakeholder interviews, the provision of multiple copies was understood to be a requirement of the Department.


Source 4: Member sheds and other potential applicants’ surveys

Office Holder surveys (Round 1 and Round 2)

The Round 1 group consists of 220 respondents and the Round 2 group consists of 169 respondents. A total of 163 respondents completed the Round 1 survey (74%) and 74 (44%) completed the Round 2 survey, however questions related to the NSDP were posed early in the survey. Those who provided qualitative responses may have identified more than one theme within their response. Therefore, number of responses refers to the prevalence of themes identified, regardless of the number of individuals who responded to the specific question. No individual spoke to more than two themes.

Survey Questions:

Has your Shed ever applied to AMSA for funding through the NSDP?

Yes No

213 (68.27%) 99 (31.73%)

If ‘no’, why not? (71 respondents)

I didn’t know about the grants

I didn’t have the time to apply

I didn’t have the resources to apply

I didn’t have enough information about the

application process

21 (23.60%) 6 (6.74%) 2 (2.25%) 20 (22.47%)

Other reason(s) for not applying? – 40 responses (44.94%)

Funded elsewhere (14, 35.00%) A total of fourteen responses indicated that the reason for not applying for funding through the NSDP was due to the fact that their Sheds are funded by non-AMSA parties. The sources of funding were not always identified, however those that were, were named as state, city, community, internally and funded by the lottery. The remaining responses described that their Sheds were currently well-equipped and did not require Grants.

Don’t have a shed yet (9, 22.50%) The second most common response detailed that the office holders do not yet have a physical Shed or had very recently obtained a physical Shed. This was reported to be due to not yet having a property or lease, were in the process of moving, or were using shipping containers in the interim.

Restrictive criteria (6, 15.00%) Six responses discussed how the funding criteria prevented the office holders from applying for funding through the NSDP. Within these, the responses identified that operating out of another organisation’s site and the Shed’s requirements are not aligned with the criteria (e.g. running costs).

Application concerns (5, 12.50%) The process of the application was also identified by respondents as a reason for not applying for the funding through the NSDP. Some feedback regarding this detailed that the application was too complicated, impersonal and demanding. Other comments highlighted that there was not enough time to obtain quotes for the application and secondly that there would be a small likelihood of success.

Waiting for appropriate time (3, 7.50%) Three office holders indicated that they were waiting to apply for funding at the right time for their Shed. For example, Sheds were waiting for a specific project that required funds and anticipating growth of the Shed before applying.

Other (3, 7.50%)


One of the remaining responses indicated that the office holder was not aware what the NSDP program was and communicated that they do not have influence in this area. Another stated they were unsure of the question. One stated bureaucracy within their Shed being a barrier.

__________________________________________________________________________________

Table 4.6.1.

Ratings of Office Holders: (127 respondents) < 10% not shown on graphs

In your view, how could the NSDP process be improved?

Note. Specific suggestions for improvement were identified only once unless otherwise indicated in brackets.

Application process (67) The most commonly described improvement for the NSDP process was in relation to the application process itself. The key response in relation to the application was that of the timeframes associated with it (n=24). Some responses specified that the issue was with the process from applying to receiving funds being lengthy (n=7), as this makes it difficult to budget, poses challenges for the cost and availability of equipment and presents safety risks that, in one case had forced the Shed to buy the equipment themselves to improve safety. Other office holders identified that they would like quicker responses (n=5) in terms of speeding up the announcement of successful applicants once decisions have been made. Finally, the decision-making process was suggested to be reduced in time (n=13) as there are too many people involved in the process and the advertised timeline should be met. Three suggestions were offered in order to assist with this – one, by including an expected date


of decision to be made on the application form (n=2) to ensure quotes remain valid (n=1), and two, for the decisions to be handled at the state level.

Another key response was about simplifying the process of the application (n=17), with recommendations of making the application less complicated, more user-friendly, ensuring the questions are easier to understand (n=3) and relevant (n=1), to simplify the information required, and to overall reduce the size of the application (n=4). Additionally, there were responses that identified a need for the application forms to be streamlined (n=7). This was particularly in relation to the repetitive nature of subsequent application forms. One respondent suggested for the inclusion of a question of whether the Shed’s details have been changed to help mitigate this issue. Some responses identified that the process of the NSDP Grants would be improved with a reduction in the number of application copies required (n=4) as it poses a financial burden on the Shed. One response stated getting quotes for small items from local businesses was difficult, and photocopying and posting the documents was expensive.

Some further suggestions included allowing greater time for Sheds to prepare their applications (n=3), removing the quote component of the application entirely (n=2), moving to an online application, for the process to be more receptive to the needs of small Sheds, greater equity of Grants, application assistance online or via telephone and prevention of allowing the process to be monopolised by powerful state bodies.

Communication (36) Improved communication was suggested in various domains and for various reasons by respondents. General suggestions of more information (n=14) and better communication (n=5) were frequent. Some described how this could be improved, with suggestions for more regular emails and newsletters, readily accessible information, regular maintenance of AMSA website, providing quicker responses, direct contact with the Shed secretary/nominated member online, greater clarity of the assessment process and when funds will be available and communication of delays. Some called for more specific improvements where face-to-face support, greater support for remote Sheds (n=2), application assistance for non-computer savvy members, direct contact from the NSDP to the Sheds, greater liaison with WAMSA, and an increased advertising of Grants (n=3) were discussed. Finally, other Shed office holders identified that they would like to see the NSDP process more available in Victoria and a suggestion for more discussion.

Funding (23) The third most prevalent suggestion for improvement of the NSDP process related to funding. The primary comment in this area was a recommendation of further funding available to Sheds (n=15). Three of these responses provided further explanation as: to keep up with the number of Sheds, provide greater funding in each category and add more types of funding. Similar suggestions included that of a start-up Grant for all new Sheds (n=3) (e.g. $1000) to assist with the initial registration, paperwork, insurance and communication set up.

Additional recommendations of improvement in relation to funding were around how the funds were to be allocated, suggesting equality of funds distribution, allocating repair Grants to ensure hand saws and blades can be sharpened for safety, an increased Grant pool for smaller Sheds, and “if the cost of projects that qualify for NSDP funding exceeds the cap on funds available in one year, could funding be staged over consecutive years based on a project timeline.” A final suggestion was for assistance in putting pressure on local councils to aid in providing suitable Shed space.

Criteria (19) The criteria associated with the Grants were the fourth most discussed improvement. Some comments related to communication of the criteria itself and the basis of assessment (n=3), with one suggestion of informing member groups that they qualify for certain Grants. Other comments spoke to the notion of categorising the criteria differently, with suggestions of employing separate


criteria for Sheds with paid staff and Sheds that are volunteer-run and to stop funding larger Sheds entirely.

The remaining comments were in relation to how the funding is allocated. Some called for equal distribution amongst States/Sheds (n=5), whilst another shared an observation of WA Sheds being allocated the smaller Grants. Two responses identified a need to remove the political (including local member) process involved in the funding allocation. A small group of respondents called for an objective basis of deciding on funds allocation as it has been observed that AMSA is currently selective as opposed to treating the submissions as equal, with suggestions of it to be merit-based, and a preference of refraining from favouring Aboriginal and Torres Strait Islander Sheds. Finally, some office holders felt as though the Grant applications were a waste of time due to being informed that their postcodes were too affluent (n=3).

Feedback (4) A smaller theme of comments related to the absence of feedback given to Sheds relating to the NSDP process. Two office holders expressed a preference for feedback around why the Shed failed to receive funding, whilst another two respondents suggested feedback on their Shed’s application progress throughout the process.

Other comments (5) Some responses provided some extra insight without fitting into a specific category. Three comments spoke to the fact that they had not received funds after one attempt. Office holders provided extra information to reflect their perspective that they did not read the guidelines very well and that the committee was under the impression that the Shed would not be successful.

An alternative suggestion for improvement was to provide a greater emphasis on the need to help members stay committed to maintaining the Shed’s activities to ensure members are active. Another expressed a concern for accountability and discussed the preference for the process to maintain its simplicity as bureaucracy is resented in their community.

Other – Satisfied as is, no comment (100) The remaining responses did not directly respond to the question in some way. There were three key areas that were discussed within these responses. A commonly described response (n=44) was that the question was not applicable to the office holder responding. However, some described why (n=10). These reasons were due to having not been successful in gaining funds after applying (n=5), having only applied once (n=1), being new to the system (n=1), that the process was handled by other Shed members (n=2) and that the responders were not aware of the NSDP (n=3).

Another common response spoke to satisfaction with the current NSDP process (n=40). Within this, the feedback ranged from office holders being quite happy with the process, to stating that it is okay and no improvements required. Comments that identified a specific benefit to the current process were the satisfactory service despite living 200km from a major city, the application process being better than others, support from AMSA staff and fair distribution of available limited funds. One response indicated satisfaction with the process, even though they were unsuccessful in their last application.

The other key reason for not directly responding to the question was due to respondents not having had experience with the process as they have not applied before (n=12) or the process was still ongoing (n=1). Some respondents did not identify why (n=4), whereas three reasons were described by other respondents. New Sheds had not yet applied (n=4), and other Sheds did not require NSDP funding (n=4).

evaluation of the australian men’s shed...

Documents