Bruno CurvalePresident of the ENQA board

Bologna expertHead of international affairs at AERES

29th of May 2009

Vilnius

ENQA experiences with external reviews of agencies

- Challenges -

Outline of the presentation• Introduction

• I - European Quality AssuranceMajor steps

• II - The ESG and the ENQA membership criteria• Similarities and differences

• III - ENQA experiences• ENQA coordinated reviews• The work of the board

• IV - The issue of consistency• Interpretation• Notions• Cultural dimension

• V - Improvements and remediation• Debriefing• Training

There are many challenges related to:

• The context

• The objectives of ENQA

• The link between the ESG and the ENQA membership criteria

• The complexity of the two stages evaluation process

• The nature of the judgements to be made by the review panels and the ENQA board

• The methodology itself

I - European Quality AssuranceMajor steps

Bologna process and quality assurance developments

1999Bologna

2001Prague

2003Berlin

2005Bergen

2007London

European cooperation in quality assurance

Primary responsibilit

y of HE institutions for quality

European Standards and Guidelines

European RegisterEQAR

Cooperation of QA agencies and HE institutions

E4 Group

1998 Reco. 98-561-CE 2006 Reco. 2006-143-CESource : Colin Tück, E4, B. Curvale

04/2008

2009LeuvenLouvain-la-Neuve

Evaluation of EQAR

II - The ESG and the ENQA membership criteria

ENQA objectivesSimilarities and differences between the ESG and

the ENQA criteria

ENQA objectives

• Promotion of agencies

• ENQA full membership as a recognised sign of professionalism

• A single external review of agencies for ENQA membership and EQAR listing

Differences and similarities between the ESG and the ENQA Criteria

ENQA criteria

▸ Principle based or rules based? Agencies shall follow the ESG and have to comply with the membership criteria

▸ Correspond to part 3 of ESG ▸ Integrate the part 2 of ESG (and by consequence the part 1 of the ESG)▸ Occasionally associate Standards (ENQA criteria 1)▸ Occasionally mix Standards and guidelines but keep the requirement

in the conditional (ENQA criteria 4)▸ Occasionally go further than the standard or the guidelines (ENQA

criterion 3, ENQA criterion 7)▸ Occasionally split Standards in different criteria (ENQA criteria

6i and 6ii)▸ Have additional requirements (ENQA criteria 8i, 8ii, 8iii)

But are fundamentally identical to the letter and the spirit of the ESG, unless it is the opposite

III - ENQA experienceENQA coordinated reviewsThe work of the board

The experience

• 7 ENQA coordinated reviews organised so far (ANECA, AQA, AQU, ACSUG, AGAE, ARACIS, EUA-IEP)

• 2 to be done (SKVC for sure and ECCE)

• 24 agencies (among 39 full members) are already externally reviewed (AAC, AQA, FHR, VLHORA, NEEA, EVA, ACQUIN, AQAS, ASIIN, FIBAA, GAC, ZEvA, HAC, HETAC, NQAI, NVAO, NOKUT, PKA, AGAE, ANECA, AQU, NAHE, OAQ, QAA)

• 8 candidates members are preparing to the review

The types and the coordination of the external review process

Type A (ENQA membership criteria / ESG)Type B (also other aspects)

- statutory functions (national missions, …)- ECA Code of Good Practice- special context (e.g. bi-national character, NVAO)- follow-up on previous review…

Co-ordination • national• ENQA (+ other non-national)

Sources: Tibor Szanto, Barcelona 17/03/09

The work of the panel

Is the agency…- fully compliant- substantially compliant- partially compliant- non-compliant

… with the criterion/ESG standard

Overall judgement not required (but can be made if the panel wishes to do so)

Sources: Tibor Szanto, Barcelona 17/03/09

The work of the board

To answer the question: Can it follows the analysis and thejudgements made by the panel and decide to grant the full ENQA membership.

What the board expects from the report:- ESG Parts II and III fully covered- Each criteria discussed separately- Structure (under each criteria):- evidence („description”, „facts and data”)- analysis („consideration”)- conclusion (judgement on compliance)- recommendation (if any)

Sources: Tibor Szanto, Barcelona 17/03/09

IV - The issue of consistencyConsistency of judgement

Consistency of judgement is a challenge

For the panels and for the ENQA board because of:

Situations have to interpreted

Notions used are difficult

Language issues

Cultural dimension

Implicit knowledge

V - Improvements and remediationDebriefingTraining

The experience is gathered

Well organised review processes are determinantInternal processes and external processes are equally important

ENQA is learning from what is done by the panelsDebriefing/brainstorming meeting for the training of experts for agency reviews, Barcelona, Spain, 17 March 2009

ENQA has started to train external reviewersTraining session/workshop for experts for agency reviews, 21-22 May 2009, Paris, France

The experience is shared

ENQA develops its documentation and makes it available to all:The Guidelines for national reviews of ENQA member agenciesThe Briefing pack for review panel members of ENQA coordinated reviewsThe Principles for ENQA coordinated reviews

are now merged in one single document:

The Guidelines for external reviews for the purpose of ENQA membership

ENQA will continue to develop the methodology of external reviews

The board has now a “book of precedents”, regularly updated by the ENQA secretariat

Thank you for your attention

bruno.curvale@orange.frwww.enqa.eu

…

4. Enable higher education institutions active within their territory to choose among quality assurance or accreditation agencies in the European register an agency which meets their needs and profile, provided that this is compatible with their national legislation or permitted by their national authorities

5. Allow higher education institutions to work towards a complementary assessment by another agency in the European Register, for example to enhance their international reputation.

…

Hereby recommend that member States:

The European parliament and the council of the European Union…

Extract of the Recommendation of 15 February 2006 on further cooperation in QA in HE

All agencies which comply substantially with the European Standards and Guidelines for Quality Assurance (ESG) can be admitted to the Register.

Substantial compliance with the European Standards and Guidelines is to be evidenced through an external review by independent experts. Such a review is coordinated either by a national authority or another organisation that is independent from the quality assurance agency under review.

Full ENQA membership, being also based on substantial compliance with the ESG, will normally constitute satisfactory evidence for inclusion in the Register.

EQAR Criteria

Extract from the EQAR website http://www.eqar.eu

Standard 3.3

Guidelines 3.3

Standard 3.1

ESG 3.1, 3.3 / ENQA Criterion 1: Activities

Extract from the Application form for membership of ENQA

“The Agency undertakes external quality assurance activities (at institutional or programme level) on a regular basis. These may involve evaluation, review, audit, assessment, accreditation or other similar activities and should be part of the core functions of the Agency. In undertaking its activities, the Agency should take into account the presence and effectiveness of the external quality assurance processes described in Part 2 of the European Standards and Guidelines (ESG”

Guidelines 3.5

Standard 3.5

ESG 3.5 / ENQA Criterion 4: Mission statement

Extract from the Application form for membership of ENQA

“The Agency should have clear and explicit goals and objectives for its work, contained in a publicly available statement. This statement should describe the goals and objectives of the Agency’s quality assurance processes, the division of labour with relevant stakeholders in higher education, especially the higher education institutions, and the cultural and historical context of its work. The statement should make clear that the external quality assurance process is a major activity of the Agency and that there exists a systematic approach to achieving its goals and objectives. There should also be documentation to demonstrate how the statement is translated into a clear policy and management plan.”

Standard 3.4

“The Agency should have adequate and proportional resources, both human and financial, to enable it to organise and run its external quality assurance process(es) in an effective and efficient manner, with appropriate provision for the development of its processes, procedures and staff.”

ESG 3.4 / ENQA Criterion 3: Resources

Extract from the Application form for membership of ENQA

Standard 3.8

Guidelines 3.8

“The Agency should have in place procedures for its own accountability. These procedures are required to include the following:

i. a published policy for the assurance of its own quality, made available on its website;

ii. documentation which demonstrates that:……

ii. a mandatory cyclical external review of its activities at least once every five years which includes a report on its conformity with the membership criteria of ENQA. Documentation could here either indicate that a review has taken place or is planned.”

ESG 3.8 / ENQA Criterion 7: Accountability procedures

Extract from the Application form for membership of ENQA

“are expected“ in the ESG guidelines

Review AND evaluation of the review

AgencyProcedures, “behaviour”

Q Standards

Review of the agency(ESG 3, 2 and 1)

+ specific ENQA criteria

Purposes

Evaluation of:- the review process- the review outcomes

National regulationsSpecific objectives

Inclusion in EQAR

ENQA full membership

External review

ENQA board

Register committee

1 2