report of external evaluation and revie · drainlaying, concrete roofing, tiling, metal roofing ......

Report of External Evaluation and Review Plumbing, Gasfitting and Drainlaying Industry Training Organisation Ltd

Not Yet Confident in ITO performance

Not Yet Confident in capability in self-assessment

Date of report: 17 October 2012

2

Contents

Purpose of this Report................................................................... 3

Introduction ................................................................................... 3

1. ITO in context ...........................................................................................3

2. Scope of external evaluation and review..................................................5

3. Conduct of external evaluation and review...............................................6

Summary of Results ...................................................................... 8

Findings ...................................................................................... 12

Recommendations ...................................................................... 24

Appendix ..................................................................................... 25

MoE Number: 8122

NZQA Reference: C05057

Date of EER visit: 19, 20, 21 October 2011, 2 February 2012

3

Purpose of this Report The purpose of this external evaluation and review report is to provide a public statement about the industry training organisation’s (ITO) performance and capability in self-assessment. It forms part of the accountability process required by Government to inform investors, the public, trainees, communities, employers, and other interested parties. It is also intended to be used by the ITO itself for quality improvement purposes.

Introduction

1. ITO in context Name of ITO: Plumbing, Gasfitting and Drainlaying Industry

Training Organisation Ltd (PGDRITO)

Gazette recognition date: 28 February 2008. The recognition for this ITO will expire on 20 October 2012.

Gazette coverage: “Pursuant to section 5 of the Industry Training Act 1992, full recognition is given to the Plumbing, Gasfitting and Drainlaying Industry Training Organisation Limited for a period of five years with effect from 20 October 2007 to set standards at Levels 1 to 8 on the National Qualifications Framework for the plumbing, gasfitting and drainlaying, concrete roofing, tiling, metal roofing and cladding manufacturing industries.”

Number of qualifications registered on NZQF:

Seven plumbing, gasfitting, and drainlaying

Two roofing

Number of standards registered on NZQF:

141 plumbing, gasfitting, and drainlaying

27 roofing

Number of trainees: 2,858 (1,958 plumbing and gasfitting trainees (1,050 plumbing, 908 gasfitting); 440 drainlaying; 460 roofing)

(Based on PGDRITO Annual Report 2010)

Number of Standard Training Measures (STMs):

1,637 (1,280 Industry Training; 357 Modern Apprenticeships)

(Based on Tertiary Education Commission 2011 contracted funding approvals)

4

Number of staff: 22

Number of registered workplace assessors:

PGDRITO has a register of workplace assessors, most of whom are institute of technology and polytechnic (ITP) tutors who assess off-job and on-job training.

Distinctive characteristics: PGDRITO arranges training in plumbing, gasfitting, drainlaying, and roofing through seven ITPs and one private training establishment (PTE).

Recent significant changes: PGDRITO has:

• extended the number of provider contracts to improve regional opportunities for training

• increased the number of regional training advisors who support trainees with literacy and numeracy needs

• introduced an off-job training system whereby plumbing and gasfitting apprentices attend a week-long block course every 17 weeks

• developed a skills recognition process which enables experienced plumbers and gasfitters to gain their national certificate

• introduced business and management qualifications for those who manage plumbing, gasfitting, drainlaying, and roofing businesses and workforces

• joined the Built Environment Training Alliance, a network of building sector ITOs set up to support the rebuild of Canterbury after the recent earthquakes.

Previous quality assurance history:

PGDRITO was previously quality assured by audit in March 2007 and did not meet seven of 13 requirements in relation to workplace assessors and national external moderation. The main reason for the requirements not being met was that the quality management system was not clearly documented and used by all assessors.

An investigation of an external complaint in 2011 resulted in NZQA finding that the PGDRITO- supplied assessment materials used by an ITP did

5

not fully assess the unit standards on a block course. NZQA required PGDRITO to review assessment records and arrange reassessment for trainees. Relevant matters related to this are included in this EER report. The ITO and block course providers are expected to complete their response to the complaint findings by the end of April 2012.

Other: On 1 July 2005, the industry training organisation separated from the Master Plumbers, Gasfitters and Drainlayers NZ Inc to become the Plumbing, Gasfitting and Drainlaying Industry Training Organisation Limited, encompassing plumbing, gasfitting, drainlaying, and roofing.

2. Scope of external evaluation and review The scope of the external evaluation and review of PGDRITO included the following focus area:

• Governance, management, and strategy.

This is not a mandatory focus area for ITOs, but its inclusion was agreed by the ITO given the organisation’s history and the importance of its coverage.

The other focus areas were:

• Relationship with the Plumbers, Gasfitters and Drainlayers Board

This includes the partnership between the ITO and the registration body, the Plumbers, Gasfitters and Drainlayers Board, and a key initiative for the development and implementation of a unified national certificate and registration and certifying system. This would integrate the national certificates with the board examinations and eliminate two separate assessment systems for industry and registration of plumbers, gasfitters, and drainlayers.

• New off-job delivery initiatives

The ITO has several initiatives under way to improve the participation and retention of trainees, as well as completions. These initiatives include developing off-job training delivery options to better suit employers and workplace priorities and the quality of off-job training and assessment. New common assessment tasks and other training materials have been developed in collaboration with industry and providers for use by contracted providers and by employers and their apprentices. Trials of induction workshops for all new plumbing, gasfitting, and drainlaying trainees, and a “12 term model” whereby plumbing and gasfitting apprentices

6

attend a week-long block course every 17 weeks, have been held and are being evaluated by the ITO.

• Collaboration, cooperation, and coordination

This includes collaboration and cooperation with other ITOs through the Built Environment Training Alliance, which the chief executive of PGDRITO formerly chaired. The alliance’s strategic goal is to ensure a highly productive building and construction sector with a workforce with the right skills, and with employers able to make best use of those skills. The cluster approach is intended to ensure a coordinated approach to skills development and to reduce fragmentation and duplication of effort. It is intended to be part of the Construction Industry Productivity agenda led by the Department of Building and Housing and to connect with the Canterbury earthquake rebuild.

This focus area also includes modern apprenticeships coordination, regional training advisors, and literacy and numeracy.

Audit of ITO assessment and moderation practices

During and subsequent to the EER visit, NZQA conducted an audit of the ITO’s assessment and moderation practices to determine whether gaps in the off-job assessment of a small group of apprentices and the recording of their sources of evidence identified by NZQA in response to an external complaint were systemic. The audit covered the contracting arrangements for off-job training, monitoring of trainee pathways, moderation results and processes, and the ITO, PTE, and ITP processes for reporting trainee results.

The audit has contributed to the EER. A separate audit report has been sent to the ITO.

3. Conduct of external evaluation and review All external evaluation and reviews are conducted in accordance with NZQA’s published policies and procedures. The methodology used is described fully in the document Policy and Guidelines for the Conduct of External Evaluation and Review – Supplement for Industry Training Organisations available at: http://www.nzqa.govt.nz/providers-partners/registration-and-accreditation/external-evaluation-and-review/policy-and-guidelines-for-eer-ito/introduction/. The ITO has had an opportunity to comment on the accuracy of this report, and any submissions received are fully considered by NZQA before finalising the report.

An NZQA lead evaluator led a team of three other evaluators. An NZQA advisor also attended. The team spent three days at PGDRITO’s head office site in Wellington. The evaluators interviewed the chief executive officer, qualifications and quality systems manager, national operations manager, finance manager, projects manager, the PGDRITO board chair, and the Plumbers, Gasfitters and Drainlayers Board chair and a board member. Team members also spoke to

7

training advisors, assessors, moderators, advisory group members, qualifications developers, industry representatives, suppliers, and trainees.

A range of the ITO’s documents was reviewed including: governance and management documents such as strategic and business plans and reports, quality management system policies and procedures, assessment and training materials, evaluations and feedback, and the PGDRITO website.

Submissions received from the ITO after the completion of the site work were fully considered by NZQA before the finalisation of this report.

8

Summary of Results

Statement of confidence on ITO performance NZQA is Not Yet Confident in the performance of Plumbing, Gasfitting and Drainlaying ITO Ltd.

PGDRITO’s core purpose is to provide industry leadership on skills and training matters and current and future skill needs, and to design standards and national qualifications to meet these needs, and to arrange the delivery of industry training.

The ITO’s strategic and business plans and various development projects indicate that it mostly manages these statutory functions effectively, although the EER team had concerns about several issues: the length of time taken to achieve some important outcomes such as the development of roofing resources and reaching agreement with the Plumbers, Gasfitters and Drainlayers Board about the registration of graduates; some inconsistencies in arrangements for training, such as incomplete contracts and providers having to use their own initiative as to the standards to be assessed; and the moderation system. The ITO is aware of the risks of not managing provider contracts effectively, whereby the arrangements for training are not clear and are not properly monitored and can result in errors in assessment and reporting.

PGDRITO, its board, and the registration board are working towards a closer and more collaborative relationship and are clearly committed in principle to providing a streamlined qualifications and registration system, and training that is accessible to a greater number of apprentices and their employers.

Since 2005, the ITO has invested in the upgrade of existing qualifications for currency with legislation and industry practice, and in developing new qualifications at levels 4 and 5 for improved career pathways. Such initiatives, at different stages of development and implementation, should contribute to a significant increase in the number of qualified and registered plumbers, gasfitters, drainlayers, and roofers (roofers are licensed separately by their own body). PGDRITO’s participation in such alliances as the Built Environment Training Alliance is also important for ensuring a productive building and construction sector with the appropriate skills.

The ITO acknowledges that there is still more to be done in improving training, assessment, and moderation resources and practices for all trades, and particularly roofing. It has made significant improvements to increase participation and completions in plumbing, gasfitting, and drainlaying, in association with new versions of qualifications. Following the introduction of new off-job delivery systems, initial indications point to positive improvements in completions.

In 2010, approximately 45 per cent of all trainees completed their programmes. This compared with a national median of 47 per cent for all ITOs and ranked PGDRITO as twenty-first out of 38 ITOs. Provisional data for 2011 shows that 78 per cent of trainees completed their programmes, which would rank PGDRITO

9

second out of 32 ITOs. In 2010, the proportion of credits achieved by trainees compared with the credits trainees were enrolled in was 69 per cent, which exceeded the national median of 53 per cent and ranked PGDRITO eighth. Provisional data for 2011 shows 107 per cent achievement, which would rank PGDRITO third out of 32 ITOs.

These results affect the numbers of trainees trained and qualified, so are important to other industries, businesses, residential properties, the rebuild of Canterbury, and the New Zealand economy. However, the educational performance indicator (EPI) data is an indicator of only one part of the ITO’s performance and there are gaps in some of its systems. Hence, the evaluation team supports the ITO’s initiatives but it is not yet confident in its performance. For clarity, a selection of the reasons for this are grouped according to the three statutory functions defined under the Industry Training Act 1992 section 6, and as summarised by the Tertiary Education Commission1:

1) Set skill standards for their industry(ies)

• The length of time taken (by the ITO) to develop qualifications, systems, and resources is considerable and puts employers and their trainees at risk.

2) Develop arrangements for the industry training to be delivered, including arrangements for monitoring and assessing trainees

• While the mix of off-job and on-job training is appropriate, it is not yet clear that the two fully support each other, or include sufficient employer engagement in determining workplace competence

• Thirty per cent of level 4 plumbing graduates do not gain their registration

• Gaps in assessment and recording are not always picked up by the quality management system

• The research and self-assessment information is not comprehensive enough to make value judgements on the benefit of outcomes for Māori, Pasifika, and female.

3) Provide leadership within the particular industry(ies) on matters relating to skill and training needs

• The future skills and training needs of the ITO’s industries are not fully understood (while several strategic documents were presented at the EER, how well the ITO is using these to meet this leadership function was not clear or demonstrated)

• It does not appear that all management and staff share a common vision, purpose, and direction

1 http://www.tec.govt.nz/Learners-Organisations/Industry-Training-Organisations-ITOs/ITO-Governance/

10

• There is a lack of clear evidence showing how successfully the strategic and business plans have been informed by self-assessment findings, or how well they have been implemented

• Information is not always readily available to enable identification of issues and good decision-making.

Statement of confidence on capability in self-assessment NZQA is Not Yet Confident in the capability in self-assessment of Plumbing, Gasfitting and Drainlaying ITO Ltd.

The EER team believes that the evidence presented at the EER was insufficient to show that the ITO knows how well it is performing and the reasons why or why not. Self-assessment is of inconsistent quality and used inconsistently by the ITO to understand its performance and bring about improvements. Improvements are occurring inconsistently, and the informational base on which they are being introduced is limited or unclear. While there are documented processes for self-assessment (a self-assessment planner and worksheet, and a process developed by an external consultant), there is no evidence that these detailed processes are followed comprehensively or how they result in improvements.

Self-assessment was not part of a coherent and comprehensive approach throughout the ITO. The lack of a comprehensive database for the moderation system compromises the ITO’s ability to interpret and analyse assessment issues and trends and use this data for improvement. It is commendable that PGDRITO compares itself with the Industry Training Federation’s report: High performance in ITOs, Incorporating ITO Practices and Models Survey. However, the value of the ITF’s report as a self-assessment and benchmarking tool would be greatly enhanced if all staff were involved in determining how well their ITO is performing on an ongoing and regular basis using the report’s most relevant characteristics.

PGDRITO needs to become more independent and self-sufficient through better self-assessment based on the analysis of quality data (both quantitative and qualitative), internal and external discussions and reflections on what the data means, and consideration of the effectiveness of any subsequent changes. This would enable self-assessment to become an essential part of the culture of the ITO and the practice of every staff member, which would in turn contribute to ongoing improvements.

Reasons why the EER team is not yet confident in the ITO’s capability in self-assessment also include:

• Systematic needs analysis of its consultation for the review of standards, qualifications, and resource development is not carried out

• Changes and innovations incorporated into operations are not evaluated sufficiently to determine their effectiveness

11

• The data collection and reporting components of the moderation system are not sufficiently comprehensive to be used effectively

• Some important processes have not been in operation long enough for their outcomes and effectiveness to be fully apparent

• The self-assessment scope could use employer, trainee, and other stakeholder feedback more effectively.

Notwithstanding the above, the evaluation team thinks that there is sufficient evidence of individual and organisational self-assessment to convince it that PGDRITO has the capacity to improve its capability in self-assessment.

12

Findings2

1.1 How well does the ITO understand and meet the needs of industry, learners and government?

The rating for performance in relation to this key evaluation question is Adequate.

The rating for capability in self-assessment for this key evaluation question is Adequate.

PGDRITO is responsible for four industries that are vital to New Zealand and to the Canterbury rebuild. It is aware of the importance of significantly increasing the number of qualified and skilled plumbers, gasfitters, drainlayers, and roofers. Such people are always in demand and many more than usual will be needed to undertake the rebuild of Canterbury while not reducing the level of services to the rest of New Zealand. The ITO’s participation in the Built Environment Training Alliance is important to ensure a productive building and construction sector with the appropriate skills to meet the needs of Canterbury and New Zealand.

PGDRITO was the seventeenth largest ITO in terms of Tertiary Education Commission investment in 2010. At the time of the EER, it had recently had its 2011-2013 Investment Plan accepted by the Tertiary Education Commission, and funding had been approved to the end of 2012. This indicates that PGDRITO continues to meet the Government’s strategic and economic objectives for industry training.

Consultation for the review of standards, qualifications, and resource development, and trialling of resources and systems for plumbing, gasfitting, drainlaying, and roofing training programmes was evident, although there was no evidence of ongoing systematic needs analyses or evaluation.

PGDRITO expects that its new database will provide a more effective tool for recording and analysing diverse sources of evidence of skill needs and for arranging and monitoring off-job training and assessment contracts with providers. It should also enable the organisation to evaluate the effectiveness of changes and ultimately improve the quality of training, assessment, and moderation for all trades.

In regard to the arrangements for training, the ITO appears to understand the immediate needs of its industries. PGDRITO has surveyed key stakeholders, including employers and apprentices over 2008-10 to obtain information about training effectiveness, employer characteristics and engagement, and off-job delivery options. It has improved working relationships with key stakeholders in order to design programmes to meet these needs. However, the evaluation team is concerned about the length of time being taken to develop and implement new

2 The findings in this report are derived using a standard process and are based on a targeted sample of the organisation’s activities.

13

standards and qualifications and resources and the associated risks for the industry of not having tradespeople competent in current practices. The reliance on providers using new industry codes and regulations while new standards and qualifications and resources are being developed suggests potential risks and it is unclear whether the ITO’s monitoring and moderation takes such practices into account. The lengthy transition (three to four years) between versions 3 and 4 of the plumbing and gasfitting qualification (during which providers continued to use inadequate training and assessment materials for apprentices enrolled in version 3), contributed to the EER team’s concerns.

The ITO’s understanding of short and long-term workforce training needs could be enhanced by more in-depth research, future scanning, and systematic needs analyses which could include comparisons with countries such as Australia, the UK, and Germany. There is some evidence of this in the ITO’s strategic, investment, and business plans and annual and consultants’ reports, and in the ITO’s use of demographic information sources and predictions of the numbers of qualified tradespeople needed for the rebuild of Canterbury. While information gathered from surveys of trainee and employer satisfaction, industry penetration, wider data from wider outcome indicators, labour market surveys, demographic / ethnicity / gender information on participation in the relevant industries is evident, it could be improved, as could its use. For example these reports and documents could be analysed and reviewed to extract specific data for estimating future skills needs across the ITO industry sectors and regions.

The ITO has made some significant improvements to its arrangements for training to increase participation and completions in plumbing and gasfitting by introducing a one-week block course system. Although, according to the ITO’s feedback, the new practices are generally winning acceptance from trainees and employers, PGDRITO still faces important challenges that should form part of its review of the block course system. For example, the ITO needs to ensure the off-job and on-job training components support each other by ensuring that trainees are prepared for and adequately assessed at the block courses, and that they have had the opportunity to complete the distance learning between block courses.

The ITO also needs to ensure that it has the best balance of on and off-job training and assessment for its industries, so that employer engagement, retention, and completions are enhanced; and that the person best equipped to assess on and off-job training does so, for instance an ITP assessor, a registered ITO assessor, or an employer. Materials and training are required to ensure employer-verifiers are fully trained to use their experience in training and assessing trainees through verification of workplace competence. These matters are under consideration by the ITO as it seeks to understand and better meet the needs of its industries, trainees, and government.

Currently, plumbing and gasfitting apprentices complete their level 4 national qualification but still need to undertake a separate examination to register and become licensed. The ITO, its board, and the separate Plumbers, Gasfitters and Drainlayers Board plan to introduce a single means of registration. A new version 5

14

of the level 4 plumbing and gasfitting qualification is planned to implement the single experiential and standards-based registration for plumbers and gasfitters. This proposal offers simplicity and clarity. However, it is not clear when this system will be in place. In the meantime, the ITO and the registration board are working closely together to bring the examination into closer alignment with the current version 4 of the qualification and to prepare trainees better for the exam. These initiatives have contributed to an increase in the exam success rate, most recently from 50 to 70 per cent (2010). Nevertheless, the EER team is concerned that a 30 per cent failure rate is still significant. The ITO and the registration board both have a responsibility to address this situation as soon as possible.

The ITO is developing a level 5 qualification which will enable licensed plumbers and gasfitters to gain the knowledge and skills required to become a certifier. This will help upskill plumbers and gasfitters and support an occupational and training pathway. The Open Polytechnic of New Zealand currently offers a programme for registered plumbers, drainlayers and gasfitters to prepare to sit the examination to become a Certifier.

The ITO has developed a skills recognition process which enables experienced plumbers to gain their national certificate faster than those who need further training and assessment in addition to their current competencies. This facilitates formal recognition of the knowledge, skills, and experience of tradespeople. A similar process for gasfitters and drainlayers is being developed.

While such initiatives reflect some understanding of the needs of the relevant industries, the basis of the understanding from which changes can be made is quite mixed and influences how well those needs are addressed. An employer survey in 2008 and an employer key stakeholder survey in 2010, as well as consultant reports and ongoing interactions with organisations such as the Plumbers, Gasfitters and Drainlayers Board, the Master Plumbers and Gasfitters Association, and the Roofing Association of New Zealand contribute to the understanding of the ITO. Nevertheless, there was limited evidence of how such information had been summarised, analysed, and discussed and how this process informed changes. There was also limited evidence of evaluations that established whether changes proved to be worthwhile improvements.

1.2 What is the value of the outcomes for employers and their trainees?



A key value of the outcomes for the government, the industry as a whole, and employers and their apprentices is that sufficient trained plumbers, gasfitters, drainlayers, and roofers are trained and employed in their respective industries and can perform quality work to the standard required. This includes meeting health

15

and safety requirements in understandably highly regulated industries. The improving programme completions and credit and qualification achievements are helping to meet the expected numbers of trained tradespeople needed in the short, medium, and long terms, including for the rebuild of Canterbury.

The value of improved arrangements for training, in particular for provider-based off-job training and assessment, is in the efficiency of arrangements that are cost-effective for employers whose employees have access to more regional training opportunities. The rotating one-week block courses every 17 weeks mean that employees have less time out of the workforce at any one time.

To further meet the needs of employers and their apprentices, the ITO is actively embedding literacy and numeracy in its resources and assessment materials and increasing the number of regional training advisors who support trainees with their literacy and numeracy needs, where the ITO identifies (usually at the point of entry) who needs their help. Version 4 of the plumbing and gasfitting qualification includes a mathematics standard to support numeracy skills acquisition. Such initiatives not only help individual trainees learn and achieve their qualification and registration, they also help to develop a more literate and numerate workforce.

Since version 4 of the plumbing and gasfitting qualification was introduced in 2009, employer surveys, trainee and employee feedback, and evaluation team interviews with assessors and moderators confirm the industry acceptance or support of improvements in training and assessment design and delivery. The ITO points to increased retention and completions arising from the programme improvements. This applies also to other industries, including roofing, where development is in an earlier phase than the others. The deficiencies in roofing assessment and moderation are being addressed, with new resources under development.

The ITO questions the value of outcomes of pre-trade courses for industry. It is concerned that only 6 per cent of students completing level 2 pre-trade courses at ITPs go on to gain apprenticeships, and that there is the potential for ITPs delivering these courses with higher equivalent full-time student funding to withdraw resources from ITO-contracted block courses to focus on pre-trade courses. The ITO also thinks that it would be better for apprentices and their industries if it had fuller operational responsibility for the Modern Apprenticeships programme rather than independent modern apprenticeship coordinators as is currently the situation. While it accepts that the decision for any changes would have to be made by the Tertiary Education Commission, the ITO considers that changes to funding and other policies would enhance its ability to add value.

There may be a higher degree of self-assessment happening at the governance level of the ITO board than across and within the ITO itself, but the board also needs to ask questions about or challenge the reports it receives on a regular basis and to support better self-assessment information underpinning the rationale for the changes it is asked to approve. It is critical that the ITO knows why initiatives are proposed and how effective and successful such initiatives are when they are delivered and how relevant and valuable their outcomes are. Robust assessment

16

and moderation and sharing findings that can inform good practice are vital to all assessment and self-assessment. So are ongoing employer, key stakeholder, and trainee feedback. The ITO also needs more research and better self-assessment information, including demographic and ethnic and gender data, to enable it to ensure that the value of its outcomes for employees and their employers applies to male and female trainees of all ages as well as to Māori and Pasifika trainees, and are clearly identified, relevant, and important at all times. Such information would support increased participation as well as completions.

1.3 How well do trainees achieve?

The rating for performance in relation to this key evaluation question is Good.

The rating for capability in self-assessment for this key evaluation question is Good.

Improved programme and qualification completions, and the proportion of credits achieved by trainees compared with the credits trainees were enrolled in, referred to in the Summary of Results earlier in this report, are reflected in the increasing number of credits achieved: from 79,000 in 2008, to 85,000 in 2009, and 100,000 in 2010. The 2010 PGDRITO annual report forecast that credits would increase to 120,000 in 2011, but this was raised to 250,000 in November 2011 based on progress up to October 2011. Such credit achievement is related to a reduction in “overdurations” of qualification completions. The ITO expects that the introduction of one-week block courses and other initiatives such as the level 4 (version 4), level 5, and business and management qualifications will mean that how well trainees achieve will improve in the future as the qualifications (and related resources) provide pathways and upskilling as the uptake of these qualifications increases.

From 2008 to 2010, 13-14 per cent of PGDRITO’s trainees were consistently Māori and around 1 per cent Pasifika, which is lower than the national population (17 and 8 per cent respectively). There was, therefore, an opportunity in the 2011 business and investment plans to increase these participation proportions to 16 per cent and 4 per cent respectively in 2011 and close the gap in programme and qualification completion between all trainees compared with Māori (which is lower by up to 7-8 per cent), and Māori compared with Pasifika (which is another up to 7-8 per cent lower again). The success of the ITO for Māori and Pasifika for participation and completions is not clear. Provisional 2011 data shows Māori participation of 14 per cent and Pasifika under 3 per cent, but completion rates are not yet available.

The new database should help the ITO to better centralise, capture, and share information (including assessment and moderation information), and contribute more effectively to analysis, reflection, changes, and evaluation of their success. It should enable the ITO to better track individual trainee progress and help provide the focused support that facilitates success. It is too soon for evidence of the effectiveness of the quality of such contributing processes and related achievement outcomes to be fully apparent.

17

1.4 How effective is the training arranged by the ITO?

The rating for performance in relation to this key evaluation question is Good.


The ITO arranges off-job training and assessment through a network of regional providers and is committed to improving these partnerships for the benefit of apprentices and their employers. Recent improvements include workshops for tutors and collaboration over the development of standardised training and assessment resources.

The training arranged by the ITO overall is effective. This is demonstrated by improved participation and programme, credit, and qualification completion rates compared with previous years and other ITOs, and is supported by employer, trainee, and provider feedback.

The new off-job delivery initiatives appear to contribute to the effectiveness of training arranged by the ITO. Feedback from experienced industry assessors and moderators confirms the “huge improvements” in assessment resources and in the consequent understanding and better relationships between all participants in the training programme.

The increase in participations and completions in plumbing and gasfitting has resulted in a 35 per cent increase in block courses held in 2011. (Some of these courses enable trainees who have been in the system for some time to catch up.) The new block course practice is popular with trainees and employers alike, according to the ITO, contributing to almost a doubling of national certificate completions despite the plumbing and gasfitting trainees who were expecting to complete in 2011 not being able to do so until 2012. Distance learning now includes assessment based on unit standards in the qualifications and is no longer a qualification requirement without any meaningful assessment.

The new system provides better continuity in training and assessment for apprentices. Employers like a one-week block course every 17 weeks because it gives them certainty and less disrupted service provision. However, the new block course training system still faces important challenges that should form part of the ITO’s review. Inclusion of teaching, learning, and assessment in one week is demanding on the apprentices. The ITO needs to ensure that off-job training and practical on-job training components and experience are complementary and support trainee achievement, particularly because all assessments (including distance learning) are conducted by ITP tutors within the week-long block courses. The ITO needs to ensure the capability of all participants and to engage and use employers more.

The ITO has recently extended the number of provider contracts to improve regional opportunities for training and now arranges training with seven ITPs and one PTE. The EER-related audit found that while PGDRITO had a separate contract with each provider, the standards to be assessed were either omitted or

18

incomplete. However, in the sample audited this does not appear to have affected the coverage or effectiveness of the training and assessment provided as the providers knew what standards they needed to assess and did so using the assessment materials provided by another ITP, their own, or those of the ITO. PGDRITO accepts that it could better manage such contracts and is working more closely with ITPs jointly to achieve and evaluate the contracted outcomes.

PGDRITO has the responsibility of assuring the quality of the training and assessment it arranges (and pays for), and for the validity of results it submits to NZQA. In future, the ITO intends to use performance-based contracts which will obligate the ITO and the provider to monitor contracts and trainee progress more closely, thereby enhancing quality assurance by the ITO and the provider.

The mix of off and on-job training and assessment is appropriate for plumbing, gasfitting, drainlaying, and roofing. New packages include resources that are designed to reduce gaps between off and on-job training. The ITO is currently focused on embedding its literacy and numeracy initiatives in the plumbing, gasfitting, and drainlaying qualifications in that order and has yet to do so for roofing.

A wider scope for self-assessment needs to include and use better ongoing feedback on employer, stakeholder, and trainee satisfaction. The effectiveness of the training that the ITO arranges should serve and be effective for all trainees regardless of ethnicity (including Māori and Pasifika), gender, or age, but the ITO’s self-assessment does not show that this is the case.

1.5 What is the quality of the assessment being undertaken towards industry standards and qualifications?

The rating for performance in relation to this key evaluation question is Good


The quality of assessment design, resources, and practice for qualifications that have been revised relatively recently is high, and the move towards a single plumbing and gasfitting registration system with industry approval, embedded literacy and numeracy, the introduction of one-week block courses, and the meaningful integration of distance learning assessments into programmes are all important developments for PGDRITO. The impact of these changes in assessment practice needs to be evaluated for further learning by the ITO and to confirm the role of the changes in improvement in performance.

The assessment of on-job and off-job training is complementary as industry standards require a high level of practical knowledge and skills that are most effectively learned and assessed when they are integrated as much as possible. The effectiveness of the introduction of one-week, off-job block courses competencies in this respect are yet to be fully evaluated.

19

The introduction of version 4 of the level 4 plumbing and gasfitting qualification from 2009 provided the ITO with the opportunity to not only revise and improve national standards and qualifications, but also to improve the related delivery systems and resources. The ITO has worked closely with contracted providers and industry to produce improved assessment design and materials. The new resources now include pre-moderated and approved common assessment tasks (CATs), with built-in feedback and monitoring. Pre-assessment moderation and the use of CATs will reduce the risks associated with the use of the training and assessment materials that varied in validity and quality in version 3 of the plumbing and gasfitting qualification. Providers wishing to use their own materials will still need to have them pre-moderated as well as their post-assessment decisions.

The training and assessment materials for the new drainlaying qualification are task-based, with unit standard matrices and guidelines, and include resources for employers as well as trainees. Once evaluated, they will be used as a model for further development by the ITO.

Moderation indicates how effective and fit for purpose assessment is. The ITO’s moderation showed that the quality of the assessment being undertaken towards industry standards and qualifications in the older version (3) of the plumbing and gasfitting qualification was quite mixed. Based on the information supplied by the ITO, it can be seen that of the 82 moderation events, 42 per cent of the materials submitted was listed as being at the “national standard but requiring some modification”, and 47 per cent as “not at the national standard” (the summary did not include MIT moderation events).

In addition, the ITO’s moderation strategy as part of transitioning from version 3 to version 4 contradicted its own Accreditation and Moderation Action Plan and focused on the moderation of perceived high-risk standards only. As a result, more than 200 version 3 plumbing and gasfitting apprentices were placed at considerable risk of not being fully assessed and qualified. This gap is now being addressed with the full cooperation of the ITO and the providers concerned. If the ITO’s assessment and moderation system had been more robust, and information and individual actions better captured and shared with all providers, the ITO would have been more likely to have identified the seriousness of the situation in the first place and been able to deal with it.

PGDRITO has the responsibility, as the standard-setting body, and the capability to act on individual training and assessment issues. For example, in consultation with industry and moderators it determined that some standards, since revised in version 4 of the plumbing and gasfitting qualification, were not achievable in their current format, either because they were outside the work experience of most apprentices and their employers’ scope of work, and/or were very specialised. When it was realised that not all gasfitting installation could be assessed on-job as well as off-job, based on experience (because of the limited occurrences of some highly specialised installations), the ITO advised all providers that training and assessment could be simulated instead.

20

The ITO has shown that it can manage complex assessment and moderation activities when needed, as it did when it developed guidelines for filling the gaps in the training and assessment of the 2010 and 2011 plumbing and gasfitting trainees. The gaps were mainly in the assessment materials used and the recording of assessment evidence which included naturally occurring evidence in the workplace.

With the exception of some moderation activities relating to materials for version 3 of the plumbing and gasfitting qualification, the ITO moderation system is detailed and thorough in its implementation but lacks a comprehensive reporting and data collection system that enables its coverage, information, and effectiveness to be fully understood, evaluated, and fully used.

The ITO moderation appears limited to identified individual ITP and PTE issues, which are dealt with case by case. It does not appear that individual moderation findings and advice are shared with other providers, except at annual moderator meetings. If they were, this would contribute to improved quality and consistency of assessment.

PGDRITO manages a register of 40 workplace assessors. It holds meetings of all its active workplace assessors, moderators, and providers which enables them to compare their practices and results with each other and identify improvements that can be made. The ITO wishes to involve employers more in an assessment/verifier role and acknowledges that training and development of resources is required if it is to optimise the advantages to trainees of the further involvement of such experienced practitioners.

At an organisational level, effective assessment and moderation lead to improved processes and results, improved outcomes for trainees, and better practices across each industry and between different industries. At an individual level, up-to-date records of achievement enable the trainee, the provider of off-job training, and the ITO to monitor and track individual progress and provide feedback, leading to improved outcomes. It is critical that the ITO implement and evaluate its current initiatives as soon as practicable in the interests of the industries that it is responsible for and their trainees. This includes roofing, where initiatives and the development of resources and assessment materials are at a relatively early stage compared with plumbing, gasfitting, and drainlaying. The new database should enable the ITO to aggregate quantitative and qualitative information and analyse it, thereby contributing to ongoing improvements.

Evaluation of the effectiveness of new off-job training initiatives is needed to assess the impact on the competence of industry trainees, in particular the relative merits of strategies for engaging employers in determining workplace competence, and/or contracting ITP tutors to determine this in the context of a block course. It is not clear how the ITO intends to do this. While it is clear that improvements have been made in assessment design, and this has industry support, there is no evidence of ongoing self-assessment of these developments.

21

1.6 How well does the ITO’s governance and management support the ITO to meet its statutory functions?



As stated in the Summary of Results, PGDRITO, like all ITOs, has statutory functions it is required to meet. These involve industry skills leadership, setting standards and qualifications, and managing arrangements for training.

Industry skills leadership includes PGDRITO working effectively with other industries and seven other ITOs and a PTE, and it is already involved in doing this, including in the Canterbury rebuild. PGDRITO sees the apprenticeship training system as a key to ensuring there are sufficient qualified and registered plumbers, gasfitters, drainlayers, and roofers for this work, but it is also aware of the medium and long-term need for qualified tradespeople. Significantly improved programme, credit, and qualification completions are positive indicators that this goal can be achieved.

New developments have the potential to inform similar successful initiatives, including resources and assessment materials in other areas, and particularly roofing, which needs the most development work. Effective assessment and moderation is the key to assuring the quality and management of arrangements for training. As noted, the ITO is making positive improvements to the management of contracts for training.

The launching of four business and management qualifications offers employers the opportunity to improve the productivity, profitability, and long-term future of their businesses. It also provides a further pathway for employees who have the skills, knowledge, and experience to become a more effective part of their employer’s business and/or aspire to set up their own business and become an employer themselves. This is consistent with the strategic goals of the Built Environment Training Alliance, which include employers being able to make best use of the workforce’s skills. This is a relatively recent development and the initial uptake is slow but starting to grow.

The chief executive of PGDRITO is able to articulate the ITO’s purpose and direction clearly and this understanding was shared by the national operations manager and the qualifications and quality systems manager who understand how operations relate to the organisation’s purpose and direction, including managing ongoing change. The chief executive and management staff have worked hard and successfully to incorporate the needs of key stakeholders into the strategic and operational plans of the ITO. However, it is not clear that all management and staff share a common vision, purpose, and direction, nor an embedded ongoing self-reflective and self-evaluative approach to what they do.

The EER team is concerned that the ITO management did not appear to have strategies for managing the risks associated with provider contracts, assessment,

22

and results reporting. However, the NZQA audit of the ITO’s systems found that the problems with reporting were not systemic. Nevertheless, the new provider contracts and database being put in place need to ensure that, operationally, risk has been reduced. The moderation system will need to have a robust reporting and monitoring mechanism.

PGDRITO has strategic and business plans, although it is not always clear to what extent they have been informed by self-assessment findings. These are sometimes limited to quantitative data as in the cases of programme, credit, and qualification completions. It is not clear how qualitative data and analysis, evaluation, discussion, and involvement of all staff are used in exploring the reasons for trends and how they contribute to further improvements.

PGDRITO continues to be funded by the Tertiary Education Commission and, according to its investment plan, is seeking to increase the contributions from industry. The ITO uses its funding to fulfil its statutory requirements, meet the ongoing costs of necessary technological improvements (including a new database), increase the number of regional training advisors, develop new qualifications and related resources and assessment materials, and meet other unanticipated costs. The ITO outsourced the development of a self-assessment process plan in 2010 and 2011 but this does not appear to have been widely understood, adopted, or implemented by management or staff, which has denied the ITO answers to important questions about its performance and possible improvements and the opportunity to develop an overall self-reflective and self-evaluative culture.

Better information, including research, would enhance the ITO’s performance. Nevertheless, its engagement with stakeholders to date has been sufficient to achieve a highly encouraging level of industry support overall, as indicated by increasing cash contributions from industry, which in 2011 were 27 per cent of the ITO’s total income, just above the average for all ITOs but less than the 30 per cent expected by the Tertiary Education Commission.

23

Focus Areas This section reports significant findings in each focus area, not already covered in Part 1.

2.1 Focus area: Governance, management, and strategy

The rating in this focus area for ITO performance is Adequate.

The rating for capability in self-assessment for this focus area is Adequate.

2.2 Focus area: Relationship with the Plumbers, Gasfitters and Drainlayers Board

The rating in this focus area for ITO performance is Good.

The rating for capability in self-assessment for this focus area is Good.

2.3 Focus area: New off-job delivery initiatives



2.4 Focus area: Collaboration, cooperation, and coordination



24

Recommendations There are no recommendations other than those implied or stated within the main body of this report.

25

Appendix

Regulatory basis for external evaluation and review

In 2009 NZQA introduced the evaluative approach to quality assurance in the tertiary education sector, consisting of self-assessment and external evaluation and review. This is applied in the quality assurance of programme approvals and accreditations under sections 249 and 250 of the Education Act 1989, as well as for training schemes (section 251), consents to assess against standards (section 252) AND PTE registration under Part 18 of that Act.

The NZQA Board also published policies and guidelines for the conduct of external evaluation and review of ITOs on 27 May 2010. NZQA relies on ITOs to build the self-assessment and external evaluation and review model into their infrastructure. ITOs will be evaluated on their infrastructure and use of the model in the following ways:

• Advising Ministerial recognition of an ITO under the Industry Training Act, 1992

• Accepting arrangements for monitoring and assessing of industry training under section 10 of the Industry Training Act 1992

• Registering standards on the Directory of Assessment Standards

• Awarding consents for ITOs (or their assessors) to assess against standards on the Directory of Assessment Standards under section 252 of the Education Act 1989

• Applying programme approval criteria where ITOs are course owners.

External evaluation and review is also used by NZQA as a monitoring and evaluation tool, with the outcomes of these processes informing Tertiary Education Commission decisions relating to re-recognition of an ITO. In addition, external evaluation and review reports are one contributing piece of information in determining future funding decisions in relation to an investment plan agreed between an ITO and the Tertiary Education Commission. The understandings and expectations for the implementation of ITO quality assurance are set out in a protocol to the Memorandum of Understanding between NZQA and the Tertiary Education Commission.

This report reflects the findings and conclusions of the external evaluation and review process, conducted according to the policies and criteria approved by the NZQA Board. The report identifies strengths and areas for improvement in terms of the ITO’s performance and capability in self-assessment.

External evaluation and review reports are public information and are available from the NZQA website (www.nzqa.govt.nz).

26

Information relevant to the external evaluation and review process is summarised in the publication Policy and Guidelines for the Conduct of External Evaluation and Review. Two other documents explain how the process is applied to ITOs: EER Policy and Guidelines – ITO Supplement and Evaluation Indicators for Industry Training Organisations. These documents are available at: http://www.nzqa.govt.nz/providers-partners/registration-and-accreditation/external-evaluation-and-review/policy-and-guidelines-for-eer-ito/introduction/

NZQA

Ph 0800 697 296

E [email protected]

www.nzqa.govt.nz

report of external evaluation and revie · drainlaying, concrete roofing, tiling, metal roofing ......

Documents