Download - ‘Research into Quality Assurance of Training Provision for the BANKSETA’ - November 9 th 2005 -
‘Research into Quality Assurance of Training Provision for the BANKSETA’
- November 9th 2005 -
2
The structure of this presentation is as follows …
The research map
The challenges of the BANKSETA
Quality Assurance from a process perspective
BANKSETA’s view on the definition of Quality Assurance
Selection of findings and conclusions of the research
General conclusions on Quality Assurance
Challenges Slide 3
Frame work Slide 4-12
Definition Slide 13-17
Findings Slide 18-35
Conclusions? Slide 36
- Index -
Research Map Slide 2
3
Practice 3
Practice 2
The research followed the following steps, partly in an iterative way …
- Research map -
Literature research
Challenges BANKSETA
Framework
Legislation / ETQA functions
Relevance (Filter)
‘Practice’ 1
‘Practice’ 2
‘Practice’ ..
‘Practice’ X
Initial research
Relevant Practice 1
Best Practice
+ -
In depth view
Fit to challengesImplementation
Acceptance
Information& Sources
Definition
ChallengesCountrySector
Legislation
Interviews
4
A summary of the main challenges that the BANKSETA was facing at the start of the research …
Accreditation: What is a user-friendly process for service providers and is it accessible and manageable for small providers?
Programme evaluation/approval: Should this be done and if so, what is the most effective and efficient way to do so?
Overlap: Certain scenarios result in overlaps with other SETAs with different systems in use. How can potential conflict in roles and responsibilities be avoided or resolved? Are MoUs the appropriate mechanisms?
Audits: What is the best combination of internal and external audits and how can the internal audit process be streamlined?
- Challenges BANKSETA -
5
The Primary Process: The Learning Value Chain
- Framework -
Needs Analysis
- SP / Empl -
Programme Development
- SP -
Learning- SP -
Assessment- Assessor -
Certification/Data
recording- ETQA -
Characteristics Volume driven process steps. The more programmes and the more
learners, the more often these steps have to be executed Steps are part of the Learning Value Chain. If the activities are not (well) executed this has a direct effect on the
Quality of Learning.
Pri
mary
6
Secondary Process: How do we check the Quality of the primary process?
- Framework -
Characteristics Volume driven activities. The more programmes and the more
learners, the more often these steps have to be executed Activities look directly into the steps of the Learning Value Chain. If the activities are not (well) executed this has an indirect effect on the
Quality of Learning.
Programme review ModerationLearning EvaluationInternal
External Programme review ModerationLearning Evaluation
Needs Analysis
- SP / Empl -
Programme Development
- SP -
Learning- SP -
Assessment- Assessor -
Certification/Data
recording- ETQA -
Secon
dary
Pri
mary
7
The Assessor
The ETQA registers Assessors.
Tertiary Process: The capacitating of the entities involved in the primary and secondary processes. Who do we need
to capacitate?
- Framework -
Internal
External Learning Evaluation
Needs Analysis
- SP / Empl -
Programme Development
- SP -
Learning- SP -
Assessment- Assessor -
Certification/Data
recording- ETQA -
Service Provider
The ETQA accredits, audits and empowers Service Providers
1
Programme review
2
The Moderator
The ETQA registers Moderators.
3
Moderation
Programme reviewSecon
dary
Pri
mary
Moderation
Learning Evaluation
8
Assessor
Tertiary Process: How is the right capacity of Service Providers, Assessors and Moderators ensured by the
ETQA?
- Framework -
Service Provider
Moderator
Characteristics Activities are not volume driven. Irrespective of the number of
programmes or learners, the activities will take place according to a certain frequency.
Activities do not look directly into the steps of the Learning Value Chain but check the capacity of the different entities involved in executing the steps of the Learning Value Chain and the Secondary (QA) processes connected to it.
If the activities are not (well) executed this has an indirect effect on the Quality of Learning.
Tert
iary
In take Audit Develop
Accreditation
Registration
Registration
Internal AuditExternal Audit
Monitoring visits
Workshops
?Forums
Workshops
?Forums
Workshops
9
Assessor
The total model is made up of Primary, Secondary and Tertiary Processes all working together to deliver quality
for the learner.
- Framework -
Service Provider
Moderator
Tert
iary
In take Audit Develop
Accreditation
Registration
Registration
Internal AuditExternal Audit
Monitoring visits
Workshops
?Forums
Workshops
?Forums
Workshops
Programme review ModerationLearning EvaluationInternal
External Programme review ModerationLearning Evaluation
Needs Analysis
- SP / Empl -
Programme Development
- SP -
Learning- SP -
Assessment- Assessor -
CertificationData
recording- ETQA -
Secon
dary
Pri
mary
10
The first six of the ten ETQA functions can clearly be linked to the framework and fall within the scope of the QA
research.
ETQA functions linked to framework1. Accrediting Providers2. Promoting Quality amongst Providers3. Monitoring provision4. Evaluating assessment and facilitating moderation amongst providers5. Registering assessors6. Certificating learners
ETQA functions outside the framework1. Co-operation with relevant moderating bodies2. Recommending new standards or qualifications (or modifications to
existing standards and qualifications) to SGBs 3. Maintain a database4. Submitting reports to SAQA
- Framework -
11
All the ETQA functions are linked to three streams of the Training & Quality Assurance process
- Framework -
Primary process“from Needs Analysis
till Certification”
Needs Analysis & Programme
Development
Learning Assessment Certification
Secondary Process
“Assuring the Quality of the
actual process“
Int
Ext
Tertiary process
“Assuring the Quality of the
different entities
involved”
Entity Intake process Audit process Empowering
Service Provider
Assessor
Moderator
1 Accrediting Providers
2 Promoting Quality amongst Providers
3 Monitoring Provision
4 Evaluating assessment and facilitating
moderation amongst providers
4 Evaluating assessment and facilitating moderation amongst
providers
5 Registering Assessors
6 Certificate learners
12
Legislation indicates the following guidelines for the Quality Assurance processes …
- Framework -
Primary process“from learning till
certification”
Needs Analysis & Programme Development
Learning
Must be OBET.Must be according to
NQF standards.
Assessment
Guidelines are given by SAQA:
- assessor requirements-principles of assessment- assessment process
Certification
ETQA must keep database acceptable to
SAQA and have means of preventing fraud
Secondary Process
“Assuring the Quality of the
actual process“
IntNo relevant legislation.
Guidelines are given to check Assessment as well
as to do moderation internally
N/A
Ext
SP needs to proof ability to develop,
deliver and evaluate prog which culminate in spec reg std and
qualifications.
No relevant legislation.
SAQA may appoint moderating bodies to
ensure credible assessment decisions. ETQAs
need to have a system to manage assessments.
N/A
Tertiary process
“Assuring the Quality of the
different entities
involved”
Entity Intake process Audit process Empowering
Service Provider
QMS requirement, link qualification, relevant P&P,
resources, accr. by one ETQA, prov. accreditation
possible
Legislation does not specify, only that ETQAs should monitor and audit achievements in general
No relevant legislation.
Assessor
Must have successfully completed assessor training
and have subject matter expertise.
ETQAs need to have a system to manage
assessments.
No relevant legislation.
Moderator
Must have successfully completed moderation training + knowledge of
subject area.
ETQAs need to have a system to manage
moderation.No relevant legislation.
13
Currently the BANKSETA has the following Quality Assurance processes in place …
- Framework -
Primary process“from learning till
certification”
Needs Analysis & Programme Development
Learning
(Done by 32 SPs8562 registered learners currently on database)
Assessment
(4424 assessors on the database. Assessment
guide available for assessors.)
Certification
Certain parts devolved to providers.
Secondary Process
“Assuring the Quality of the
actual process“
IntProgramme Review informed by QMS
Training evaluation informed by QMS
Internal moderation based on SPs QMS (2% policy,
10% practice)
Ext
Evaluation of selection of programmes by site-visits (only format, e.g.
OBE)
Reviewing of internal moderating reports (part of site-visits) + in case of
problems
Tertiary process
“Assuring the Quality of the
different entities
involved”
Entity Intake process Audit process Empowering
Service Provider
‘Preliminary and formal’Criteria: similar to SAQA, focus on D&D control +
internal audit
Requires yearly internal audit by SP, External Audit only if
necessary based on appeals + Annual report on moderation and
assessment
Workshops, Consulting
AssessorCompetent against US 115753, have technical
expertise
ModeratorCompetent against US 115759, experienced registered assessor
14
The more we focus on the BANKSETA itself, the more the ‘general knowledge’ needs to be extrapolated to the
BANKSETA
Quality Assurance
Quality Assurance in training
Quality Assurance for SAQA
Quality Assurance for the BANKSETA
- Definition -
Scope
Precision
Rese
arch
Dis
cuss
ion
15
The General Definitions of Quality Assurance ….
ISO 9000:2000 “Part of quality management focused on providing confidence that
quality requirements will be fulfilled.”
Six Sigma “Work done to ensure that Quality is built into work products, rather
than Defects.” This is by (a) identifying what “quality” means in context: (b) specifying methods by which its presence can be ensured; and (c) specifying ways in which it can be measured to ensure conformance.
Q-Africa 2004 Conference Paper “The sum of activities/elements that assure the quality of products and
services.”
- Definition -
16
Quality Assurance for training is considered …
CHE “The underlying objective of the HEQC with regard to quality assurance
is: to ensure that institutions effectively and efficiently deliver education, training, research and community service which are of high quality and which produce socially useful end enriching knowledge…”
Engineering Council “Quality assurance: in general, a process which examines the quality of
a process and its product; when applied to an education and training programme, evaluating the programme and its assessment against standards using defined policies and procedures.”
Higher Education - Australia “(The) systematic management and assessment procedures adopted to
monitor performance and achievement and to ensure achievement of specified quality or improve quality.” (Harman & Meek, 2000, p. 11)
- Definition -
17
The view of SAQA and the SETAs …
SAQA’s definitions “Quality Assurance means the sum of activities that assure the quality
of products and services at the time of production or delivery. Quality assurance procedures are frequently applied only to the activities and products associated directly with the goods and services provided to external customers”
“Quality Assurance means the process of ensuring that the degree of excellence specified is achieved”
“Implicit in SAQA’s implementation of the quality system is the understanding that quality assurance, quality management and accreditation are not things or products; rather, quality is a process.” (Quality is an ongoing event)
- Definition -
18
What does this mean specifically for the BANKSETA?
The extrapolation to a definition of quality assurance for the BANKSETA:
= = = = = =
“Quality assurance refers to the integrated processes that evaluate the provision and
outcomes of education, training and development, so that compliance with SAQA
requirements is achieved and systems for continuous improvement are in place for
the benefit of learners.”
= = = = = =
- Definition -
Learning- SP -
Assessment- Assessor -
CertificationData
recording- ETQA -
Pri
mary
Programme Development
- SP -
Needs Analysis
- SP / Empl -
In ScopeOutside Scope
19
The following sources gave input to the findings
Literature General literature on Quality Assurance Skills Development Legislation (National & International) SAQA documentation ETQA Guidelines and Policies
Interviews A total of 28 persons were interviewed representing the following
groups: Employers, Service Providers, ETQA managers, SAQA officials and independent experts
Practices 5 ETQAs were researched 3 International Practices were researched (Aus, New Zealand,
Malaysia/Scotland)
- Findings: Primary Process -
20
Practices of the Primary process
- Findings: Primary Process -
Needs Analysis
- SP / Empl -
Programme Development
- SP -
Learning- SP -
Assessment- Assessor -
Certification /
Data recording
- ETQA -
Pri
mary
Programme Approval by ETQAs1) Yes, including SME and review committee2) Yes, mainly as part of Accreditation3) Yes, mainly as part of Accreditation4) Yes, mainly as part of Accreditation5) Yes, mainly as separate process, no SME needed
Programme Approval InternationalAustralia: Use of nationally endorsed Training Packages
+ Programme ≠ Material
Material ∈ Programme
21
The view of experts on the Primary processes
- Findings: Primary Process -
Needs Analysis
- SP / Empl -
Programme Development
- SP -
Learning- SP -
Assessment- Assessor -
Certification /
Data recording
- ETQA -
Pri
mary
Employers:“Business Impact will inform the quality better than assessments”
Service Providers: “Too much focus on Assessment and not on Training Needs affects the performance negatively”“SPs tend more toward measuring business impact of training to improve themselves”
ETQAs: “There is a risk in the Employer – Assessor relationship”
Employers: “Huge problem with time that assessments take”
Scope issue
Scope issue
ETQAs: Programme approval is …- Very important- Not so important- SME needed- SME not needed
Independent Experts:“We don’t know how to measure the impact of the NQF and with that the ETQA function”
22
Programme Approval is a gate keeper for the end result, Subject Matter Expertise can be applied depending on the
context …
- Findings: Primary Process -
Safety
Service
Compliance
Banking Sector
Turn around time from sector in case of incorrect content of programme
Pote
nti
al Im
pact
of
err
ors
Thoroughness needed in
Programme
Approval
Possibilit
y of relying on se
lf regulatory fu
nction of th
e
market
Outcomes based
Aligned with US
Aligned with US according to SME
Material correct according to SME
E X A M P L E S
Approval by committee
23
Conclusions on the Primary process
- Findings: Primary Process -
Needs Analysis
- SP / Empl -
Programme Development
- SP -
Learning- SP -
Assessment- Assessor -
Certification /
Data recording
- ETQA -
Pri
mary
“Garbage in is Garbage out”
“Performance Management is needed”
“Needs analysis falls outside the scope”
1
2
3
24
Way forward on the Primary process
- Findings: Primary Process -
Needs Analysis
- SP / Empl -
Programme Development
- SP -
Learning- SP -
Assessment- Assessor -
Certification /
Data recording
- ETQA -
Pri
mary
Suggestions
1. Introduce a gate keeper in the form of Programme Approval:- Minimise the use of Subject Matter Expertise- Take samples based on ‘smart’ selection
2. Do further research into whole process of assessments. Is it possible to address the challenge of the sector?
3. Use the Performance Auditing that will be introduced by SAQA to start measuring the performance of the ETQA function from a client perspective (the learner). This will enable continuous improvement of the ETQA processes.
1 2 3
25
Practices of the Secondary process
- Findings: secondary process -
Moderation by ETQAs1) Is done incidentally, policy to minimise it.2) Is not done3) External moderation is done
Moderation InternationalAus: No secondary processes are performed
Programme review ModerationLearning EvaluationInternal
External Programme review ModerationLearning Evaluation
Secon
dary
Combined with Approval
Internal checksMostly directly or indirectly covered in Accreditation / QMS criteria
Does not existIncidentally considered
26
Experts on the Secondary process
ModerationLearning EvaluationInternal
External ModerationLearning Evaluation
Secon
dary
- Findings: secondary process -
Independent Experts: “Too much emphasis on assessment and moderation, we should be putting systems in place”
Independent Experts: “Moderation and Program Evaluation should be built-in systems and part of self evaluation”
Experts: “Quality Assurance Mechanisms are being set up for the sake of compliance rather than to improve Quality”
SAQA: “The main degrees of freedom that occur where overlap between ETQAs take place, lie in the Secondary Processes”
Service Provider: “Let me pass the (accreditation) mark and let me then run my own processes”
Programme review
Programme review
27
Service Provider Qualification Sector
Normal CasesCase 1 I a, b, c A Case 2 II i, j, k BCase 3 III x, y, z C
Special CasesExtra-sectoral I a B Extra-qualification I i AExtra-sectoral/qualification I I i BExtra-sectoral/qualification II I x B
- Findings: secondary process -
The overlap between different SETAs leaves us with four combinations with many more possible applications (1) …
28
The overlap between different SETAs leaves us with four combinations with many more possible applications (2) …
Service Provider QualificationSector/ETQA
Special Cases1. Extra-sectoral I a B 2. Extra-qualification I i A3. Extra-sectoral/qualification I I i B4. Extra-sectoral/qualification II I x B
ETQA activities 1 2 3 4
TertiaryAccreditation A A A A
Auditing A A A A
SecondaryProgram Review ? ? ? ?
Ext Moderation ? ? ? ?
Primary
Certification A B B C
DATA – NLRD A B B C
Data – DOL B A B B
- Findings: secondary process -
Are these activities done? Who is doing them? Can they be outsourced to another
ETQA?
29
Approval
Conclusions on the Secondary process
“MoU challenge = secondary processes”
“Programme Review = Secondary Process”
“Programme Approval = Primary Process”
“Promotion of a Culture of Quality?”
1
2
3
Programme review ModerationLearning EvaluationInternal
External Programme review ModerationLearning Evaluation
Secon
dary
- Findings: secondary process -
ETQA activities 1 2 3 4
TertiaryAccreditation B B B B
Auditing B B B B
Secondary
Program Review ? ? ? ?
Ext Moderation ? ? ? ?
Primary
Certification B A B B
DATA – ELRC
Data – DOL B A B B
° Freedom
Programme Development
- SP -
Learning- SP -
Policing? / Redoing? / Empowering?
Programme Review
30
Way forward on the Secondary process
ModerationLearning EvaluationInternal
External ModerationLearning Evaluation
Secon
dary
- Findings: secondary process -
Suggestions
1. Minimise the effort put into Secondary Processes
2. Rather focus on Tertiary Processes that enable Service Providers to perform internal Secondary Processes
3. Work in conjunction with SAQA towards clarity concerning the degrees of freedom that occur around the Secondary Processes in case of overlap between ETQAs
Programme review
Programme review
31
Practices of the Tertiary process
- Findings: Tertiary Processes -
Assessor
Service Provider
Moderator
Tert
iary
In take Audit Develop
Accreditation
Registration
Registration
Internal AuditExternal Audit
Monitoring visits
Workshops
?Forums
Workshops
?Forums
Workshops
ETQAs - AccreditationProvisional option often usedAll different categories / Criteria within SAQA framework
International AccreditationAus: larger scope, includes advertising and marketing Can, Scotland: incl. e.g. Marketing and Needs Analysis
ETQAs - AuditingInternal Audit not always applicableMostly aligned with AccreditationMixed with Secondary Processes
International AuditingAus: 1-1 alignment with Accreditation
32
Experts on the Tertiary process
Assessor
Service Provider
Moderator
Tert
iary
In take Audit Develop
Accreditation
Registration
Registration
Internal AuditExternal Audit
Monitoring visits
Workshops
?Forums
Workshops
?Forums
Workshops
- Findings: Tertiary Processes -
Service Provider: “There is not always a clear link between Accreditation Criteria and how it can positively affect business processes”
Service Provider: “Internal Audits can be a helpful tool to create an internal platform for change and performance improvement”
Service Provider: “Accreditation is still a cumbersome process”
ETQAs: “Accreditation is the key-element to ensure performance”
Independent Experts: “Accreditation process is often not fast enough, results in bottlenecks”
General: “It is possible to get accredited and not perform”
33
Although alignment is important, the Demming wheel suggests an accent on the “Act” phase when performing
external audits
- Findings: Tertiary Processes -
Accreditation
=
Int. Audit Ext. Audit
Checking the results of the Primary Process
Taking Corrective Actions
Plan
Do
Check
Act
Primary Processes
Needs Analysis
- SP / Empl -
Program Development
- SP -
Learning- SP -
Assessment- Assessor -
Certification /Data
recording- ETQA -
Planning the Primary Process
What is the procedure for Programme Development?
What is the business planning in terms of Programme Focus?
What is the procedure for Programme Evaluation?
What are the corrective actions based on Plan, Do,
Check?
34
The categories for accreditation/auditing are informed by the Primary and Secondary processes where the SP is
involved.
- Findings: Tertiary Processes -
Programme review ModerationLearning EvaluationInternal
Needs Analysis
- SP / Empl -
Programme Development
- SP -
Learning- SP -
Assessment- Assessor -
Certification/Data
recording- ETQA -
11 22
33 44
5566 77
88
Secon
dary
Pri
mary
SA
QA
99
……
Secto
r S
pecifi
c
xx
e.g. overhead processes
35
Accreditation and Audit Criteria could be summarised and structured as follows …
General QMS requirements, e.g.- Management responsibility- Follow-up systems
Specific Policies, Procedures and Processes, e.g.- Core
Programme Development, Programme Review (+ requirements)Provision, Learning Evaluation (+ requirements)Assessment, Moderation
- OverheadHR (+ requirements) Financial (+ requirements)…
Static Criteria, e.g.- Admin and Financial Capacity, Legal Entity, …
- Findings: Tertiary Processes -
36
Conclusions on the Tertiary process
Assessor
Service Provider
Moderator
Tert
iary
In take Audit Develop
Accreditation
Registration
Registration
Internal AuditExternal Audit
Monitoring visits
Workshops
?Forums
Workshops
?Forums
Workshops
“thorough processes are needed”
“simple and aligned processes”
1
2
- Findings: Tertiary Processes -
37
Way forward on the Tertiary process
Assessor
Service Provider
Moderator
Tert
iary
In take Audit Develop
Accreditation
Registration
Registration
Internal AuditExternal Audit
Monitoring visits
Workshops
?Forums
Workshops
?Forums
Workshops
Suggestions Establish rhythm of Accreditation, Internal Audit and External Audit Align accreditation 1-on-1 with Audits and site visits Focus on a Culture of Quality that goes beyond compliance. Look for
‘real’ evidence in the “Act” phase The strength of the sector suggests that Provisional Accreditation is
currently not a necessity. Rather focus on empowerment.
- Findings: Tertiary Processes -
38
The following bullet points remain open for discussion …
Can one size fit all?
Is a sector specific model the route to follow?
Given the degrees of freedom, is it possible to make sensible comparisons?
How close is Best Practice to Common Practice?
- Conclusions? -