Review of the ISTA Accreditation System: evaluation of the questionnaires completed by the laboratories and the Designated Authorities
Rita Zecchinelli
Report of the ISTA Accreditation Review Working Group
Background
During the Ordinary Meeting 2012 in Venlo (NL), the membership of ISTA approved a motion jointly proposed by the ISTA Executive Committee and the Designated Authorities from Australia and New Zealand. The motion proposed a review of the audit process, in order to improve its cost effectiveness, considering the overall cost to laboratories, without reducing standards. During the Ordinary Meeting 2013 in Antalya (TR), two questionnaires were presented, one for the Designated Authorities, and one for the laboratories. Both questionnaires have been distributed and responses received.
19.06.2014 – R. Zecchinelli
ISTA Accreditation Review Working Group
Composition (approved by the ISTA ECOM on 25.01.2013)
Rita Zecchinelli (Chair, ISTA ECOM member) Joël Léchappé (ISTA President) Craig McGill (ISTA Vice-President) Masatoshi Sato (ISTA ECOM member) Ronnie Don (ISTA Honorary Life member) Rasha El-Khadem (Head of the ISTA Accreditation and Technical Department) Graeme Smith (representative from the Australian DA) John Randall (representative from the New Zealand DA) Piero Sismondo (representative from ISF)
19.06.2014 – R. Zecchinelli
Questionnaires
Aims of the questionnaires: - Identification of the needs of ISTA labs, DAs, other
stakeholders related to accreditation. - Collection of information on other accreditation systems
relevant for the seed testing labs. - Evaluation of the ISTA accreditation system (e.g. audits,
scope, recognition, fulfilment of needs, services, costs), even in comparison with the other relevant accreditation systems.
19.06.2014 – R. Zecchinelli
1. Questionnaire to the laboratories A) General information Answers have been received from: > 50% of ISTA member labs (106/202) ≈ 70% of ISTA accredited labs (81/120) > 80% of ISTA member countries or
distinct economies (58/71)
19.06.2014 – R. Zecchinelli
0
10
20
30
40
50
60
70
80
90
100
Sampling Traditional Tests Special test Performance based tests
% o
f IST
A a
ccre
dite
d la
bora
tori
es
Tests/Methods laboratories accredited for by ISTA Status Participant labs
Accredited only by ISTA 38 35,8%
Accredited by ISTA + other body/ies 43 40,6%
Accredited by other body/ies (no ISTA) 10 9,4%
Not accredited at all 15 14,2%
Total 106 100,0%
1. Questionnaire to the laboratories
19.06.2014 – R. Zecchinelli
A) General information: accreditation other than ISTA Answers (n) Out of % Notes
35 106 Participating labs 33 ISO accredited
71 106 Participating labs 67 ISO non-accredited (21 out of 71 state that they have planned ISO accreditation)
29 81 ISTA accredited labs 36 Also ISO accredited
6 25 ISTA non-accredited labs 24 ISO accredited
6 35 ISO accredited labs 17 With “other tests” (seed testing and/or different tests) in the scope
Answers (n) Out of % Notes
25 98 Answering labs 26 Accredited on the basis of other standards (different from ISTA and ISO)*
*When requested, only 4 labs (4%) specified a relevant standard, 13 state ISO standards, 8 do not provide information
73 98 Answering labs 75 Not accredited on the basis of other standards
19 81 ISTA accredited labs 24 Also accredited on the basis of other standards
6 25 ISTA non-accredited labs 24 Accredited on the basis of other standards
1. Questionnaire to the laboratories
19.06.2014 – R. Zecchinelli
A) General information: geographical distribution
8% 8%
4%
24% 47%
6%
3% Responses received
Africa
North America
South America
Asia
Europe
Oceania
No response
Africa 9accredited 4
not accredited 5
North America 8accredited -
not accredited 5
South America 4accredited 1
not accredited 3
Asia 26accredited 12
not accredited 14
Europe 50accredited 7
not accredited 43
Oceania 6accredited 1
not accredited 5
No response 3accredited -
not accredited 3
TOTAL 106
1. Questionnaire to the laboratories B) The needs: why ISTA Accreditation
19.06.2014 – R. Zecchinelli
0
10
20
30
40
50
60
70
80
90
100
Agree Disagree No response
Num
ber o
f lab
orat
orie
s
ISTA Accreditation is necessary because we issue ISTA Certificates
not ISTA accredited
ISTA accredited
not ISTA accredited
ISTA accredited
0
10
20
30
40
50
60
Agree Disagree No response
Num
ber o
f lab
orat
orie
s
ISTA Accreditation is necessary because it is requested by our National Authority or Administration
not ISTA accredited
ISTA accredited78%
43%
1. Questionnaire to the laboratories B) The needs: why ISTA Accreditation
19.06.2014 – R. Zecchinelli
43 out of 81 (53%) ISTA Accredited labs are also accredited by other bodies; 25 out of 43 (61%) would like to be accredited by only one accreditation body.
0
10
20
30
40
50
60
70
80
90
100
Agree Disagree No response
Num
ber o
f lab
orat
orie
s
ISTA Accreditation is very prestigious
not ISTA accredited
ISTA accredited
0
10
20
30
40
50
60
70
80
90
100
Agree Disagree No response
Num
ber o
f lab
orat
orie
s
ISTA accreditation is the most important recognition for a seed testing laboratory
not ISTA accredited
ISTA accredited
not ISTA accredited
ISTA accredited
87%
86%
1. Questionnaire to the laboratories B) The needs: why is ISTA Accreditation appreciated?
19.06.2014 – R. Zecchinelli
0
10
20
30
40
50
60
70
80
Num
ber o
f lab
orat
orie
s
In the framework of the ISTA accreditation system, I appreciate the Proficiency Test program.
not ISTA accredited
ISTA accredited
0
10
20
30
40
50
60
Num
ber o
f lab
orat
orie
s
In the framework of the ISTA accreditation system, I appreciate the competence of the
auditors.not ISTA accredited
ISTA accredited
Agree + Strongly Agree:
94%
Agree + Strongly Agree:
84%
not ISTA accredited
ISTA accredited
1. Questionnaire to the laboratories B) The needs: why is ISTA Accreditation appreciated?
19.06.2014 – R. Zecchinelli
Other reasons of appreciation: • focus on seed testing (95% agree/strongly agree) • being part of a community of seed testing labs (92%
agree/strongly agree) • receiving support from the Association (80% agree
/strongly agree). • the possibility to be accredited for performance based
approach (PBA) methods is also appreciated (74% agree/strongly agree).
19.06.2014 – R. Zecchinelli
1. Questionnaire to the laboratories
0
5
10
15
20
25
30
35
40
Num
ber o
f lab
orat
orie
s
I don’t have any reasons to criticize the ISTA accreditation system.
not ISTA accreditedISTA accredited
Have no reasons to criticise ISTA Accreditation: • 14 Company labs labs (56%) • 35 Governmental labs (51%) • 6 Private Independent labs (46%) • 55 All labs (52%)
Have reasons to criticise ISTA Accreditation: • 10 Company labs (40%) • 16 Governmental labs (24%) • 3 Private Independent labs (23%) • 29 All labs (27%)
Area No reasons to
criticise Reasons to
criticise
N° % N° % Africa (9) 5 56 4 44 North America (8) 3 38 4 50 South America (4) 3 75 0 0 Asia (26) 16 62 3 12 Europe (50) 23 46 16 32 Oceania (6) 3 50 1 17 No response (3) 2 67 1 33
Total (106) 55 52 29 27
not ISTA accredited
ISTA accredited
B) The needs: why isn’t ISTA Accreditation appreciated?
1. Questionnaire to the laboratories B) The needs: why isn’t ISTA Accreditation appreciated?
19.06.2014 – R. Zecchinelli
0
5
10
15
20
25
30
35
40
Num
ber o
f labo
rato
ries
In the framework of the ISTA accreditation system, I don’t appreciate the costs.
not ISTA accredited
ISTA accredited
Costs are a concern for 52 labs (49%) of the ISTA laboratories. Other reasons of concern: • Way the audits are conducted :12 labs (11%) labs agree/strongly agree • Audit frequency: 28 labs (26%) agree/strongly agree
0
10
20
30
40
50
60
Num
ber o
f lab
orat
orie
s
In the framework of the ISTA accreditation system, I don’t appreciate the way the audits are conducted.
not ISTA accredited
ISTA accredited
05
1015202530354045
Num
ber o
f lab
orat
orie
s
In the framework of the ISTA accreditation system, I don’t appreciate the frequency of the audits.
not ISTA accredited
ISTA accredited
not ISTA accredited
ISTA accredited
1. Questionnaire to the laboratories EVALUATION OF THE COSTS
Only very few labs provided costs of other accreditation system. 19.06.2014 – R. Zecchinelli
IN THE FRAMEWORK OF ISTA ACCREDITATION, WE PARTICULARLY DON’T APPRECIATE THE COSTS
ALL LABS ONLY GOVERNAMENTAL LABS COMPANY + PRIVATE LABS
Area Agree+Strongly
agree Area Agree+Strong
ly agree Area Agree+Strongly
agree N° % N° % N° %
Africa (9) 6 67 Africa (8) 5 63 Africa (1) 1 100
North America (8) 7 88 North America (4) 4 100 North America (4) 3 75
South America (4) 2 50 South America (3) 1 33 South America (1) 1 100
Asia (26) 10 38 Asia (15) 6 40 Asia (11) 4 36
Europe (50) 23 46 Europe (33) 16 48 Europe (17) 7 41
Oceania (6) 3 50 Oceania (3) 2 67 Oceania (3) 1 33 Blank (3) 1 33 Blank (2) 0 0 Blank (1) 1 100 Total (106) 52 49 Total (68) 34 50 Total (38) 18 47
1. Questionnaire to the laboratories B) The needs: recognition of ISTA Accreditation
19.06.2014 – R. Zecchinelli
• In general the ISTA accreditation system is widely recognized (90%). • 19 (18%) labs think that the ISTA accreditation system is not recognized
as equivalent to other accreditation system/s. • 9 do not provide further information, 5 state that ISO is more
recognized (labs from Europe and North America), 5 state that a regional or national system is more recognized (labs from North America and Asia).
• 50 labs (47%) state that ISTA accreditation is recognized only for
traditional tests, 63 labs (59%) state that ISTA accreditation is recognized for all kinds of tests.
1. Questionnaire to the laboratories C) ISTA audits: labs’ experience. General feedback.
19.06.2014 – R. Zecchinelli
0
10
20
30
40
50
60
70
Num
ber o
f lab
orat
orie
s
The ISTA audits have always provided a true picture of our laboratory.
not ISTA accredited
ISTA accredited
0
10
20
30
40
50
60
Num
ber o
f lab
orat
orie
s
The ISTA audits have always been an opportunity to improve
not ISTA accredited
ISTA accredited
05
101520253035404550
Num
ber o
f lab
orat
orie
s
The laboratory management and staff always agreed with the audit findings.
not ISTA accredited
ISTA accredited
1. Questionnaire to the laboratories C) ISTA audits: labs’ experience. Use of PTs.
19.06.2014 – R. Zecchinelli
0
10
20
30
40
50
60
70
80
90
Agree Disagree No response
Num
ber o
f lab
orat
orie
s
The results achieved by the laboratory in the proficiency tests should be taken into greater
consideration for the purposes of the ISTA audits.
not ISTA accredited
ISTA accredited
0
10
20
30
40
50
60
70
Agree Disagree No response
Num
ber o
f lab
orat
orie
s
A laboratory with low ratings in proficiency tests should be audited by ISTA more frequently than a
laboratory with higher ratings.
not ISTA accredited
ISTA accredited
But during the audits labs with high PT ratings will show a few non-conformities → 27 labs agree – 68 labs disagree labs with low PT ratings will show a larger number of non-conformities → 37 labs agree – 59 labs disagree
75% 60% 33%
not ISTA accredited
ISTA accredited
1. Questionnaire to the laboratories C) ISTA audits: labs’ experience. Frequency.
19.06.2014 – R. Zecchinelli
0
10
20
30
40
50
60
70
80
No No response Yes
Num
ber o
f labo
rato
ries
In your opinion, is the frequency of ISTA audits (one in three years) appropriate?
not ISTA accredited
ISTA accredited
In comparison with other audit systems: • 20 labs state that ISTA audits are less frequent • 15 labs state that the frequency is the same • 10 labs state that ISTA audits are more frequent
YES 75
NO 26
1. Questionnaire to the laboratories SUGGESTIONS 1/3
19.06.2014 – R. Zecchinelli
Differentiate the fees: • The fees should be tailored to the species, the country, the scope of
accreditation, the size of the lab, the number of the certificates issued by the lab.
• Organizations having labs in different countries that have implemented the same QS in all labs should pay a lower fee.
• In case of labs audited by other bodies, ISTA should use the results of those audit.
• ISTA should recognise national ISO-17025 audits. • Labs preparing PT rounds should pay a lower audit fee. • Charge the audit basic costs and separately reimburse other
expenditures. • Increase the annual accreditation fee and reduced the audit fee. • The audit fee should be transparent and cover only the costs of the
audits.
1. Questionnaire to the laboratories SUGGESTIONS 2/3
19.06.2014 – R. Zecchinelli
Review of the audit system: • Group the audits in the same region. • For good labs (no “S” findings, good PT results): audit every 4 years. • For all labs: audit every 5 years (in between: PTs). • ISTA could use the PT system + desk audits. • The audit carried out by a local independent body. • Increase the PT and reduce the audit cost of the Lab. • Take into consideration the results/rating on PTs and the duration of
accreditation (risk based approach). • If cheaper, organize one audit day with two auditors or two audit days
with one auditor. • Possibility to organize desk system audits or video audits. • Increase the number of auditors to get better representation of all
sectors.
1. Questionnaire to the laboratories SUGGESTIONS 3/3
19.06.2014 – R. Zecchinelli
Add benefits, in order to make the audit fee more valuable : • Make a particular number of ISTA certificates free by including it
in the membership fee (e.g. for the annual laboratory membership fee ISTA includes in 2 free packs of ISTA certificates. The laboratory chooses whether they want both packs to be Orange, Blue or 1 pack of each).
• Provide a checklist of requirements before the audit. • Audits with a more consulting role. • Allocate time to receive questions from the staff.
2. Questionnaire to the Designated Authorities A) General information Answers have been received from 16 Designated Authorities (≈ 23%):
2 from North America (2/2) 3 from Africa (3/10) 2 from South America (2/7) 1 from Asia (1/15) 6 from Europe (6/35) 2 from Oceania (2/2)
19.06.2014 – R. Zecchinelli
• In all countries there is at least one ISTA accredited lab.
• In 9 countries out of 16 there is at least one ISO 17025 accredited seed testing lab.
• In 5 countries out of 16 there is at least one seed testing lab accredited basing on other standards. These standards are: -national standards of the different countries, NAL (1 European country)
• ISTA accreditation is recognized by the law in 9 countries (1 in S. America, 5 in Europe, 2 in Africa, 1 in Oceania).
• ISO 17025 is recognized by the law in 4 countries (1 in S. America, 2 in Europe, 1 in Africa).
• All accreditation systems (ISTA, ISO, other standards) are recognised by popular practice in several countries.
2. Questionnaire to the Designated Authorities B) The needs: why ISTA Accreditation
19.06.2014 – R. Zecchinelli
Out of 16 DAs: • 13 out state that ISTA accreditation is necessary in order to
issue ISTA certificates. • 6 state that ISTA accreditation is necessary because requested
by the law (4 in Europe, 2 in Africa). • 14 believe ISTA accreditation the most important recognition
for a seed testing lab. • 13 think that ISTA accreditation is very prestigious. • 2 state that for seed testing labs it is necessary to be
accredited by ISTA and by another body as well. • 4 think that ISTA accreditation is not necessary at all.
2. Questionnaire to the Designated Authorities B) The needs: why ISTA Accreditation is appreciated
19.06.2014 – R. Zecchinelli
• PT programme (14 DAs). • Focus on seed testing (15 DAs). • Support provided by the association to the labs (13 DAs). • Possibility to be accredited for performance based approach
(PBA) methods is also appreciated (11 Das). B) The needs: why ISTA Accreditation is not appreciated 6 DAs don’t have reasons to criticise the ISTA accreditation
system; 4 DAs have reasons to criticise it: • 3 DAs do not appreciate how the ISTA audits are conducted. • 5 DAs do not appreciate the frequency of the ISTA audits. • Costs are a concern for 8 DAs. • 1 DA states that ISTA accreditation system is inexpensive
compared to other systems. • 7 DAs state that ISTA accreditation system gives good value for
the costs.
2. Questionnaire to the Designated Authorities B) The needs: evaluation and use of PTs results
19.06.2014 – R. Zecchinelli
• 14 DAs state that PT results should be taken into greater consideration for the purposes of ISTA audits.
• 11 DAs think that labs with low PT rating should be audited more frequently that labs with higher ratings.
• 9 DAs expect that lab with low PT ratings will have a larger number of significant non- conformities.
• 6 DAs expect that lab with high PT ratings will have a smaller number of significant non- conformities.
2. Questionnaire to the Designated Authorities SUGGESTIONS 1/2
19.06.2014 – R. Zecchinelli
Review of the audit system: • One audit every three year for the beginning (e.g. for 3 audits), then
less frequent audits if -the PT ratings is good -no disputes/complaints have been received with regards to the certificates issued by
the lab. • Modulate the audit frequency on a risk based analysis. • Define when a full audit is necessary and when a remote system
audit could be enough. • ISTA should participate as technical assessors in the ISO 17025 audit. • Recognition of audits between ISTA and ISO 17025 accreditation
systems. • Use local/regional auditors. • Use the financial tool to increase the transparency of the fees.
2. Questionnaire to the Designated Authorities SUGGESTIONS 2/2
19.06.2014 – R. Zecchinelli
Add benefits, in order to make the audit fee more valuable: • The audit should be a learning occasion for the team at the
lab. • Audits conducted in Spanish. • Add new PTs for the tests today not covered in the
programme or covered only once every several year (e.g. SHT, electrophoresis).
• Enlarge the possibility to be accredited only for sampling to non-lab institutions.
19.06.2014 – R. Zecchinelli
ISSUES RAISED FROM THE EVALUATION OF THE QUESTIONNAIRES
ISTA Accreditation Review Working Group
Frequency of the ISTA audits: should we continue to discuss this issue? Use of PT results: do the opinions appear contradictory? Costs and benefits balance: how to address the needs? Comparison with other accreditation systems: are you interested in? Are the lab accredited to ISTA and other bodies ready to provide more information?
19.06.2014 – R. Zecchinelli
FUTURE WORKS AND GOALS 1. Proposals for the revision of the ISTA accreditation system: - fees (e.g. are they reasonable basing on the evidences provided by the
use of the financial tool? can we define which expenditure is paid by which of the two fees currently paid for being accredited? should we revise the structure of the fees?)
- audit frequency (should we change it?) - audit organization (e.g. can we increase the number of local auditors?
can we define other changes/initiatives in order to improve the efficiency of the system? can we address some requests, such as provide an audit checklist?)
- proficiency test programme (e.g. can we improve the programme?) - other issues
2. Improve the recognition of the ISTA accreditation system (mutual recognition ISTA/ISO national accreditation bodies, other issues).
ISTA Accreditation Review Working Group
19.06.2014 – R. Zecchinelli
From the lab’s suggestions possibilities for changes
Model based on 2014 fees Year 1 Year 2 Year 3
Laboratory Membership (CHF) 5‘214 5‘214 5‘214
ACC: Current Model (CHF) 1‘224 1‘224 14‘224
ACC: Model based on yearly payment (CHF) 5‘557 5‘557 5‘557
• Assuming a yearly payment of the actual audit visit fee
• Pros: • Replace the large audit fee every three years and change to a
yearly fee • Reduce the year to year variation caused by the number of
audits (ISTA finances and workload)
• Can this be an issue for laboratories?
• Scope of Accreditation • Shall laboratories pay more if their scope covers many tests
and the auditors need to audit for more than one day?
• Risks when changing the system to a scope based approach: • Will labs stay with their current scope of accreditation if they
could save money removing tests they do not need for ISTA Certificates?
• More administration in Secretariat needed.
• Can this help us to be more cost effective?
From the lab’s suggestions possibilities for changes
THANK YOU FOR YOUR ATTENTION