examining quality culture part ii

E U A P U B L I C A T I O N S 2 0 11

Examining Qual i ty Cul ture Part I I :

Processes and Tools –

Part ic ipat ion, Ownership and Bureaucracy

By Andrée Sursock

Copyright © by the European University Association 2011

All rights reserved. This information may be freely used and copied for non-commercial purposes, provided that the source is acknowledged (© European University Association).

Additional copies of this publication are available for 10 Euro per copy.

For ordering information, please contact [email protected] or write to:

European University Association asblAvenue de l’Yser 241040 Brussels, BelgiumTel: +32-2 230 55 44Fax: +32-2 230 57 51

A free electronic version of this report is available at www.eua.be.

This project has been funded with the support from the European Commission. This publication reflects the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein.

ISBN: 9789078997290


Examining Qual i ty Cul ture Part I I :

Processes and Tools –

Part ic ipat ion, Ownership and Bureaucracy

By Andrée Sursock

ExAmIN INg QUAL I Ty CULTUrE PArT I I : PrOCESSES ANd TOOLS – PArT I C I PAT ION , OwNErSh IP ANd BUrEAUCrACy

4

Foreword 6

Acknowledgements 7

Executivesummary 9

PartI. Introduction 11

1.1 Scope and aim of the report 11

1.2 What is quality culture? 11

PartII. Theinterviews 14

2.1 Some facts and figures 14

2.2 The interviews: methodological considerations 15 PartIII. Contextsetting 17

3.1 Trends in higher education 17

3.2 The ten universities within this changing environment 18

PartIV. Introducingorchanginginternalqualityassuranceprocesses 21

4.1 Engaging the academic community 21

4.2 Using existing experience 23

4.3 Introducing quality assurance through staff development 23

4.4 Changing a quality assurance approach 24

4.5 Observations 25

Table of contents

5


PartV. Scopeandorganisation 26

5.1 Scope of internal quality assurance arrangements 26

5.2 Roles and responsibilities 28

5.3 Curricular committees 30

5.4 QA offices 32

5.5 Governance of quality assurance 33

5.6 Observations 34

PartVI. Toolsandprocesses 36

6.1 Collecting feedback through questionnaires 37

6.2 Collecting oral and informal feedback 39

6.3 Collecting feedback through institutional data and key performance indicators 40

6.4 Internally-organised reviews and articulation with external reviews 42

6.5 Managing academic staff 44

6.6 Observations 48

PartVII. Conclusions–structures,culturesandhistory 50

7.1 Quality assurance is about power and ideology 51

7.2 Participation and bureaucracy 52

7.3 Perceptions of academic roles 54

7.4 Institutional contexts: structures and history 55

7.5 Concluding remarks 56

References 59


6

Internal quality assurance processes are recognised as essential components of institutional autonomy, responsibility and accountability. This report – the result of the project Examining Quality Culture in Higher Education Institutions (EQC) – examines the quality assurance (QA) processes and tools that have been developed by universities in Europe. The first part of the EQC study, based on a survey questionnaire and published in 2010, showed remarkable progress in developing quality mechanisms in institutions. The goal of the second phase, which was based on 59 interviews across ten European universities, is to examine in greater depth the extent to which these mechanisms and processes have resulted in quality cultures.

The notion of quality culture is understood here as comprising (i) shared values, beliefs, expectations and commitments toward quality (ii) that are supported by structural and managerial elements and processes that enhance quality. Why is quality culture important? Simply stated, it is because it is the most effective and meaningful way that quality assurance mechanisms can ensure and improve quality levels and support a dynamic of change in universities.

One of the clearest results of this study is that the vitality and sustainability of a quality culture depends upon several internal and external factors. Internally, the university is self-confident and does not limit itself to definitions of quality processes as set by its national QA agency; the institutional culture stresses democracy and debate and values the voice of students and staff equally; the definition of academic professional roles emphasises good teaching rather than only academic expertise and research strength; and quality assurance processes are grounded in academic values while giving due attention to the necessary administrative processes. Externally, it is important that the university is located in an “open” environment that is not overly regulated and enjoys a high level of public trust. Thus, one of the conclusions of this report is that the internal and external QA processes must be viewed together in order to ensure true accountability, thus avoiding duplication of evaluations and QA fatigue.

The report illustrates, through concrete and detailed institutional examples, successes and failures in realising a quality culture. It is partly a compendium and partly an analytical discussion of processes and tools of internal quality assurance. The current pressures on universities to be effective and efficient in time of financial stringency make this report timely reading. It is hoped that it illuminates how universities can be strategic and dynamic while they continue to provide quality-assured activities.

On behalf of the project consortium, which included QAA Scotland and the German Rectors’ Conference (HRK), EUA would like to thank all parties that have contributed to the project and this report.

Foreword

Jean-MarcRappEUA President

7


This study would not have been possible without the 59 staff members, from ten universities, who agreed to be interviewed. They were promised anonymity but it is hoped that they will find themselves and their universities faithfully mirrored in the report. They were most gracious in providing time and answering questions, sometimes following up with more information via email. Their openness and interest in sharing both successful and unsuccessful experiences demonstrated their commitment to contributing to our general understanding of how best to develop and sustain internal quality assurance processes. Those who provided names of colleagues and opened doors for interviews in their institutions are particularly thanked for their help, without which this study would not have been achieved.

The Steering Committee was invaluable in providing guidance at the initial stage of this study, in discussing the preliminary findings and in commenting on an early version of this report. Its interest, commitment and enthusiasm were crucial to the success of the study.

The EUA Quality Unit gave essential assistance. Alicja Ziubrzynska organised and kept track of the interviews most efficiently. Thérèse Zhang provided ongoing support and commented on an early draft. I am particularly grateful to Tia Loukkola who read several drafts of the report in spite of time pressures; her comments were always insightful and stimulating. All three are sincerely thanked.

Finally, the European Commission, which co-funded the project under the Lifelong Learning Programme, is thanked for its support.

AndréeSursockEUA Senior Adviser

SteeringCommitteeHenrik Toft Jensen, Chair, former President, Roskilde University, DenmarkAndrea Blättler, representative, European Students’ UnionDavid Bottomley, Assistant Head, QAA Scotland, United KingdomKarl Dittrich, representative, European Association for Quality Assurance in Higher Education (ENQA)Tia Loukkola, Head of Unit, EUABarbara Michalk, Head, Section Higher Education Reform in Germany and Europe, German Rectors’ Conference (HRK), Germany Oliver Vettori, Head, Quality Management and Programme Delivery, Vienna University of Economics and Business, Austria

Acknowledgements


8

9


1. The project Examining Quality Culture in Higher Education Institutions (EQC) aimed to provide an overview of the internal quality assurance processes in place within higher education institutions across Europe and tackled the question of how they have responded to Part 1 of the European Standards and Guidelines for quality assurance (ESGs).

2. The EQC project was designed in two phases. In the first phase, a survey questionnaire addressed the seven areas included in Part 1 of the ESGs. A total of 222 institutions from 36 countries across Europe responded and the results were analysed and presented in a report entitled Examining Quality Culture – Part I (Loukkola and Zhang 2010). The report, published in 2010, showed that great progress had been made in the institutions in developing internal quality mechanisms.

3. In a second phase, 59 phone interviews were conducted with ten universities selected from the sample that responded to the survey questionnaire. The goal of the interviews was to identify the extent to which quality assurance tools and processes contributed to building a quality culture in institutions. This report is a presentation and analysis of these interviews. It is specifically focused on establishing the relationship between the formal quality assurance processes and quality culture and on illustrating – through concrete examples – good and weak practices in this area.

4. The notion of quality culture is understood here as comprising two distinct sets of elements: “shared values, beliefs, expectations and commitments toward quality” and “a structural/managerial element with defined processes that enhance quality and aim at coordinating efforts” (EUA 2006: 10).

5. The report highlights five conditions that lead to an effective quality culture:

5.1 It is important not to rely on a single quality assurance instrument, such as the student questionnaires, particularly if they shape staffing decisions (e.g., promotions). There must be a mix of several instruments to ensure good intelligence. These instruments must be related to institutional strategies and – ultimately – to academic values. Their costs and benefits must be reviewed regularly: this includes not only financial costs and benefits but also psychological aspects (e.g., do they lead to unnecessary stress or unreasonable workloads) and whether they really contribute to embedding an effective and shared quality culture, supporting the institutional strategy and providing accountability toward students and the wider public.

5.2 The most effective internal QA arrangements are those that derive from effective internal decision-making processes and structures. Having clear accountability lines and clarifying responsibilities at all levels ensure that the quality assurance system is kept as simple as possible while closing the feedback loops and this should, if anything, reduce bureaucracy by limiting data collection, reports and committees to what is absolutely necessary. It is crucial to identify who needs to know what and, furthermore, to distinguish between what is necessary vs. what would be nice to know. In addition, students and staff feel at home, first and foremost, in their faculties and departments. This argues in favour of an optimal balance between the need for a strong institutional core and a degree of faculty responsibilities, between the need for an institution-wide QA approach and some local variations in faculties.

5.3 Like external quality assurance, internal quality assurance processes are also about power. Internal quality assurance can be contested if it does not successfully engage the university community. Leadership is essential to give the initial steer and the broad frameworks of quality assurance mechanisms. Leadership should facilitate internal debate – and even tolerate dissent – in order to make sure that quality assurance processes do not end up being imposed and simply bolted on. Linked to this, the type of language used by the leadership and the QA officers in describing the QA

Executive summary


10

arrangements cannot be dismissed as trivial. The more academic and the less managerial it is, the more likely it will make inroads in the institution.

5.4 It is essential to invest in people through staff development to avoid internal quality assurance arrangements becoming punitive. It is encouraging to note the pace at which staff development schemes are growing in universities but professionally-staffed centres that support teaching and learning are still a rarity. This will require attention in the years ahead particularly because of the renewed emphasis on student-centred learning in the Bologna Process.

5.5 Both institutional autonomy and self-confidence are key factors in the capacity of institutions to define quality and the purposes of their internal quality assurance processes and to ensure that these are in line with their specific profiles, strategies and organisational cultures. In doing so, these institutions are sometimes confronted with their external quality assurance agencies’ processes, which might be at cross-purposes. It is essential that the internal and external processes are viewed together and that the higher education community – the institutions and the agencies – negotiate the articulation between the two sets of processes in order to ensure true accountability, avoid duplication of evaluations and QA fatigue.

6. The report concludes that the factors that promote effective quality cultures are that: the university is located in an “open” environment that is not overly regulated and enjoys a high level of public trust; the university is self-confident and does not limit itself to definitions of quality processes set by its national QA agency; the institutional culture stresses democracy and debate and values the voice of students and staff equally; the definition of academic professional roles stresses good teaching rather than only academic expertise and research strength; quality assurance processes are grounded in academic values while giving due attention to the necessary administrative processes.

11


1.1 Scope and aim of the report The quality, global attractiveness and competitiveness of European higher education have been

central goals of the Bologna Process. Quality assurance, however, received a relatively cursory mention in the original Bologna Declaration but, as ministers met every two years to measure progress and define mid-term objectives, the issue grew in importance, until it rose to the fore of the ministerial agenda and became one of the first policy objectives, particularly between 2003 and 2007.

The Berlin Communiqué (2003) recognised the primary role of higher education institutions in monitoring quality. This constituted the first such official acknowledgement in the context of the Bologna Process. The Bergen Communiqué (2005) adopted a text that presented three sets of standards and guidelines for quality assurance (ESGs): the first applies to higher education institutions and the others to quality assurance agencies (ENQA 2005). One of the underpinning principles of this document is the key importance of institutional responsibility for quality.

The project Examining Quality Culture in Higher Education Institutions (EQC) aimed to provide an overview of the internal quality assurance processes in place within higher education institutions across Europe and examined how they have responded to Part 1 of the ESGs. The European University Association (EUA) coordinated the project in a consortium with the German Rectors’ Conference (HRK) and QAA Scotland. A steering committee, consisting of representatives of the consortium partners, students, and quality assurance practitioners, oversaw the project.

The EQC project was designed in two phases. In the first phase, a survey questionnaire addressed the seven areas included in Part 1 of the ESGs. A total of 222 institutions from 36 countries across Europe responded and the results were analysed and presented in a report entitled Examining Quality Culture – Part I (Loukkola and Zhang 2010).

In a second phase, phone interviews were conducted with ten universities from the sample that responded to the survey questionnaire. The goal of the interviews was to identify the extent to which quality assurance tools and processes contributed to building a quality culture in institutions. This report is a presentation and analysis of these interviews. To the extent that this is the final report of a two-phase project, it links the information gathered through the interviews with the quantitative survey data collected and analysed during the first phase of the project.

1.2 what is quality culture?While both phases of the project looked at the institutional mechanisms and processes in place to

ensure quality, this report is more specifically focused on establishing the relationship between the formal quality assurance processes and quality culture and on illustrating – through concrete examples – good and weak practices in this area.

Part I. Introduction


12

The notion of quality culture is understood here as comprising two distinct sets of elements: “shared values, beliefs, expectations and commitments toward quality” and “a structural/managerial element with defined processes that enhance quality and aim at coordinating efforts” (EUA 2006: 10).

This, of course, raises the question of how to recognise that such a quality culture exists. Particular attention was paid during the interviews to identifying whether there was general agreement within an institution as to the purposes of the approach and their general acceptance and whether the quality assurance processes were bolted on or embedded. To give a specific example of how this could be identified, one interviewee stressed that his university introduced an internal quality assurance system in 2005 but:

For now, it is mechanical. There is a need to change some of the elements but, most importantly, there is a need to change minds. Academics are now accustomed to evaluations but some of them whisper: ‘I know how to teach and I don’t need students to tell me how to improve.’

If this view was confirmed by others, it could then be determined that the quality assurance processes were bolted on.

The first round of the EUA Quality Culture project identified a set of principles that promote a quality culture. These are:

• building a university community and the staff’s identification with the institution;

• developing the participation of students in the university community;

• embedding a quality culture through internal communication, discussions and devolved responsibility while understanding the resistance to change and developing strategies to overcome it;

• agreeing upon an overarching framework for quality review processes and standards;

• defining key institutional data – historical, comparative, national and international – and systematically collecting and analysing them;

• involving the appropriate external and internal stakeholders;

• stressing the self-evaluation stage as a collective exercise for the unit under review to ensure the implementation of appropriate change (this includes academic and administrative staff and students);

• ensuring a follow-up of the internal reviews: e.g., implementation of the appropriate recommendations and feedback loops into strategic management (EUA 2005: 10).

The 2005 EUA report stressed that these principles have less to do with quality assurance mechanisms than with leadership, community building and staff development schemes. Nevertheless, quality assurance mechanisms are important. Therefore, after a discussion of the methodology (Part II) and a brief context setting (Part III), this report:

• explores how internal quality assurance arrangements are introduced and changed (Part IV)

• discusses the scope of quality assurance arrangements, the distributions of roles and responsibilities and the governance of quality assurance (Part V)

• presents various quality assurance instruments, such as student and alumni questionnaires, institutional and key performance indicators, staff management and development schemes, etc. (Part VI).

13


The following diagram maps the three core parts:

These three chapters present and analyse QA instruments and processes. The primary goal of the analysis provided is to show the interrelationship between the structural/managerial elements and the existence of a quality culture.

Examples of good and weak practices are provided throughout the text to supply details that could be useful to some readers. These examples are identified clearly. Thus, the hurried reader could skip them and go directly to the analytical sections entitled “Observations” and to the concluding chapter, which analyses the specific forms that quality cultures take and links these to a set of larger issues such as democracy, trust and bureaucracy.

PartIV:Introducing or changing QA

PartV:Scope and

organisation

PartVI:Tools and processes


14

Part II. The interviews

2.1 Some facts and figuresTen universities in ten countries were included in the sample. It is important to note that the report

seeks to uncover links between quality processes and quality culture through a qualitative methodology. Thus, the choice was made to interview more people in a limited number of institutions in order to gather a more complete and reliable set of data. To the extent that the survey questionnaire was ultimately answered by one person (even if the answers were the results of internal consultations), it was hoped that the interviews would illuminate the process from a variety of institutional perspectives: leadership, staff and students.

The universities were selected from those that responded to the survey questionnaire. Selection criteria included such considerations as geographical location, institutional size, degree of maturity of the institutional quality processes, approaches to internal quality, and a mix of specialised and comprehensive institutions. As shown in Table 1, however, a geographical balance was difficult to achieve despite all the efforts deployed: repeated requests to some parts of Europe were ignored and it is unclear if this was due to linguistic concerns or other aspects came into play.

A total of 59 interviews were conducted between 12 January and 25 March 2011 (with one follow-up interview in early June). The initial contacts were generally asked to provide the names of their direct superior, a couple of students, a couple of academics, a couple of heads of departments and a couple of deans. This list was slightly modified depending on the university and the themes that arose during the first interview.

Names of potential interviewees were received from every institution, with the exception of one; here, the QA officer sent a paper that describes the QA framework. The average number of names collected was around eight and ranged from four to fourteen. Some universities had more difficulties furnishing leads than others. As a result, there is a variation in the number of interviews held in each university, as shown in Table 1 below. It is impossible to be sure about the reasons for the differences in response rates beyond noting that the more developed the internal QA processes were, the easier it was to identify appropriate interlocutors in adequate numbers in the universities.

Table 1: Number of persons interviewed per institution

Country in which the institution is located Number of persons interviewed

Austria 2

Finland 7

France 5

Hungary 6

Ireland 10

Lithuania 8

Portugal 1

Spain 9

Sweden 7

Scotland (UK) 4

Total 59

15


Most interviews were held with one person only, except in the case of four interviews in which two to three interviewees took part. Each of these collective sessions is counted as one in the table above.

The initial interviews were one hour long. Most other interviews lasted 30 minutes; some were 45 minutes long. One person was interviewed twice (and counted as one in the table above).

The range of interviewees’ statuses is as follows:

• Rector (chief academic officer): 1

• Vice rectors: 9

• Faculty administration (deans, vice deans, etc.): 15

• Heads of departments/regular academics: 11

• Students1: 6

• QA officers and some of their direct colleagues in the QA office: 11

• Others: director of Centre for Teaching and Learning; director of institutional research; executive director of operations; head librarian; former QA officer; ombudsman.

2.2 The interviews: methodological considerations

As mentioned earlier, the initial contacts (generally the QA officers) were asked to provide names of their direct superior, a couple of students, a couple of academics, a couple of heads of departments and of deans. Because many provided a long list and had contacted their colleagues before sending the names, it was difficult to extend their lists and only a few interviewees were asked to provide more names.

The advantage of asking for a list was to speed up the process of interviewing. Thus, as a counterexample, in the case of one university, one or two names were asked for after each interview, which meant progress was very slow and only a relatively small number of interviews were undertaken.

The disadvantage of relying on the initial contact was that the process became over-determined by the QA officer: his/her personal network (which provides a glimpse of the position of this office), understanding of the request (the need to speak to a range of people) and, particularly the willingness to give names of “opposing voices” to the internal quality process. In one university, where the initial contact gave a list that consisted mainly of senior administrators (and a student who did not respond), this was usefully extended by asking other interviewees for contacts.

The interviews were semi-structured and their objective was not to verify all the elements of the EQC survey but to probe into the specific experience of each interviewee and to build upon what had been learned through previous interviews in a given institution. Thus, interviewees were relatively free to take the conversation where they felt it was important and were encouraged to discuss successes and failures,

1 Many more students were contacted than responded.


16

accomplishments and challenges. Interviewees focused primarily on teaching and learning although the questions put to them were very general. As a result, this report will not discuss quality arrangements that cover research, service to society, governance and administration.

A single researcher (the report’s author) conducted all interviews, which ensured a common approach to all the conversations. About five to ten interviews in each university provided the opportunity for a reasonable degree of triangulation. The credibility of the data, however, has to be related to the size of the university, the degree of maturity of quality processes, the scope of distribution of roles and responsibilities, the degree of consensus regarding quality culture and if some representatives of a university seemed interested in participating in this study for the institution’s self-promotion (e.g., expressing such an interest in a passing statement). In this case, even greater attention was paid to the issue of triangulation.

An additional methodological point worth stressing is that, unless otherwise indicated, the generalisations made in the text refer to the sample of ten institutions involved in the study.

The introduction to the interviews stressed confidentiality, anonymity and the need to identify good and failed experiments, as well as success and obstacle factors, in order to contribute to the knowledge and development of other universities in Europe. All interviewees were told that their names would not appear in the report and would not be mentioned or quoted to their colleagues who were interviewed after them.

While most interviews were relatively easy to conduct and interviewees seemed open, a handful of

interviews were clearly difficult, either because of language skills or because the interviewee was guarded. Concerns about carrying on a conversation in English were expressed by some interviewees. In most cases, interviewees had underestimated their language skills. Some interviewees asked for questions in advance. The response was that this would be a conversation about their experience in the university rather than going through a fixed set of questions. This response seemed to provide enough reassurance.

Three institutions were interviewed in their national language (in France, Ireland and Scotland): this resulted in slightly longer interviews and had a positive impact on the scope of the data collected, particularly in France and Scotland where relatively fewer interviews were conducted.

In the other countries, one linguistic difficulty was that, across the interviews held within one institution, different words were used in reference to a single body or post: e.g., a committee might be called a board by one person, a working group by another, etc. An effort was made to try to identify these occurrences by focussing on the functions and responsibilities but this meant that some time was spent focusing on accuracy rather than collecting new data.

It is clear that the questionnaires (the first phase of the EQC project) give a partial view of the reality. It is only through the interviews that a more granular and meaningful view of the university’s quality processes can be gained. Thus, the quantitative data that appeared as positive in the questionnaire response could be tempered by the oral evidence. The open comment section of the questionnaire, however, was very helpful and was used to probe areas that the respondents had identified as being challenging.

17


Part III examines the external drivers that have led to the development of internal quality processes in higher education institutions. It locates the ten universities within an environment that has been characterised by constant change, instability and a complex set of challenges during the last decade.

3.1 Trends in higher educationIn the past decade, higher education institutions across the globe have been buffeted by a complex

set of pressures. Foremost among them is the growing importance of knowledge-led economies that have placed higher education at the centre of national competitiveness agendas. Higher education institutions are increasingly viewed by policy makers as ‘economic engines’ and are seen as essential for ensuring knowledge production through research and innovation and the education and continuous up-skilling of the workforce.

These pressures have resulted in two main European policies – the Bologna Process and the Lisbon Strategy, including the Modernisation Agenda for Universities – and have been translated into new national policies affecting principally governance, autonomy, funding, research and external quality assurance.

These fundamental changes, along with the implementation of the core Bologna reforms, are deep and significant. They have had significant impact on all the activities of universities and their partnerships with other higher education institutions and with their stakeholders and have resulted in increased emphasis on the universities’ strategic capacity and their professionalism. The changes have been time and resource consuming, especially on staff members, and require effective institutional leadership.

In addition, the current economic crisis has had a negative effect in many countries: some have had to cut their education budgets while student demand to enter higher education or to stay on for additional qualifications is growing. This, in combination with mounting pressures on public funding, has led to debate on the issue of tuition fees and free admission to higher education in some countries.

The new EU member states, however, have had access to specific streams of funding (e.g., European Structural Funds, World Bank). This has buffered them from the full impact of the economic crisis and supported the implementation of higher education reforms in which the Bologna Process played an important role as a driver and a framework for institutional change.

Finally, the rise in participation rates has led to changes in the shape and size of many higher education systems. Thus, many countries in Europe have seen a significant increase in the number of (mostly private) institutions but recent demographic declines (among other reasons) have resulted in a number of institutional mergers or bringing together several institutions under federated structures.

Part III. Context setting


18

3.2 The ten universities within this changing environment

The ten universities participating in this study were not immune from these developments and many interviewees expressed their disquiet that their institution’s survival was at stake in successfully managing this turbulent environment. They spoke of internal quality arrangements in the context of these changes.

The economic crisis has been felt most strongly in Ireland, Lithuania and the United Kingdom. As opposed to England, however, Scotland has been relatively sheltered. Nevertheless, at the time of the interviews there was a great deal of uncertainty concerning future budget cuts and the impact on Scotland of the expected rise in student fees in England. One interviewee characterised the next decade as the “age of austerity”. Another one spoke about the last two years as “having been the most challenging ever. It has been a seismic shift and a real turning point”. Several expressed the feeling that the institutions that will survive this crisis will be profoundly changed with no possible turning back to a “softer”, less managerial institution. Indeed, the impact of the current financial crisis and deep budget cuts affecting higher education are driving the senior leadership toward greater efficiency and efficacy, particularly in Ireland, Lithuania and Scotland.

The impact of the crisis, however, has been somewhat tempered by EU structural and Tempus project funding that have contributed to developing quality arrangements in Hungary and Lithuania. Universities in France and Sweden have seen no decrease in funding. Austrian, Finnish and Portuguese interviewees did not mention funding issues.

Furthermore, concerns were expressed about declining demographic trends in some countries (e.g. Lithuania), which are putting pressure on institutions to find ways to maintain enrolments, via internationalisation for instance. The issue of institutional attractiveness was felt to be particularly important in smaller countries (e.g., Lithuania and Hungary) where a relatively large number of institutions are vying with one another over a relatively small pool of students. In this competitive context, institutional reputation and quality arrangements were felt to be vital to the survival of the institution.

Given this changing environment, the development of internal quality arrangements was used to manage a change process in several institutions. Thus:

• Two of the institutions in the EQC sample were the results of mergers. In one case, the merger was fairly recent and it represented an opportunity to address merger-related issues that would have been overlooked without the development of internal QA arrangements.

• Spain and Sweden have introduced a third cycle only recently and the two institutions represented in the EQC project seized this as an opportunity to reform the curricula while embedding quality. This seemed to have been done successfully in one institution through staff development, and relatively less successfully in the other one that had decided that all new “Bologna courses” needed to be evaluated. In some quarters this seems to have led to resistance to developing new courses.

In addition, all ten countries have been implementing the Bologna Process and have introduced new policies affecting the management and governance of institutions. Table 2, based on the responses of the national rectors’ conferences to the Trends 2010 questionnaire, shows the most important policies introduced in the past ten years. It is worth noting that, although new QA frameworks are driven by the Bologna Process, this link has not always been explicit in national discussions, which explains why it is treated as a separate issue (Sursock and Smidt 2010).

19


Of particular importance to the development of internal quality arrangements, and as shown in Table 2 above, new governance and funding instruments were introduced in six of the ten countries; external QA arrangements were changed in nine of the ten countries, often with the new requirement that institutions develop internal quality processes. Thus, to give two examples, the Spanish QA agency played a pivotal role in asking all institutions to develop internal quality arrangements; the Portuguese QA agency offered an incentive: if the institutions developed good quality processes, they would only undergo a light-touch accreditation process.

As will be discussed later in this report, the requirements, focus and philosophy of a QA agency have an impact on the internal quality arrangements. The new requirements set by agencies for internal quality processes are not always linked explicitly to the ESGs; perhaps as a result, few interviewees – and then only the most senior ones – brought up the ESGs in describing the internal QA arrangements in their institution. It is also interesting to note that the issue of international or national rankings was mentioned by only two interviewees (both from Ireland) even if competitive research funding has been introduced in several countries.

In addition to the national QA agency’s activities, accreditation bodies of regulated professions (e.g., medicine) are another driver of internal quality assurance processes. They can play a positive role in raising awareness of the importance of internal quality processes in professional fields. On the negative side, their role in defining quality is often very traditional, with no interest in interdisciplinarity; they can influence university structures by stressing the autonomy of the relevant faculty. Furthermore, for universities that conduct unit reviews, the challenge is to minimise the overload caused by the combination of internal and external processes.

The Trends 2010 report noted that since the Berlin Communiqué (2003), a great deal of effort has been exerted to develop internal quality processes and that the higher education community has seized upon the Bologna agenda as an opportunity for improving teaching and learning. Indeed, the institutional responses to the Trends 2010 questionnaire revealed that for 60% of HEIs, the most important changes in the past ten years have been enhanced internal quality processes and that these developments seemed to be correlated with the international aspirations of institutions (Sursock and Smidt 2010: 18).

Thus, much has been achieved in the development of external and internal quality assurance processes, while taking into account – explicitly or implicitly – the European QA frameworks. These institutional developments have taken place as a response to a range of national, European and international change drivers but this does not mean that internal quality assurance is bolted on. They have resulted, in some cases, in an institutional ownership of internal quality assurance.

Table 2: National policy changes in the ten countries

Autonomy QA Funding Mergers/federations Others

Austria • •

Finland • • •

France • • • •

Hungary • • Research policies

Ireland • • Research policies

Lithuania • • •

Portugal • •

Spain • • Academic careers

Sweden • • •

UK-Scotland • •

Source: Trends 2010 report (Sursock and Smidt 2010)


20

The responses to the following questions are then crucial to understand how to ensure such ownership:

1. What are the conditions that lead to effective quality cultures?

2. What are the constituent elements of a quality culture?

3. What is the relationship between internal quality processes and quality cultures?

4. What is the relationship between internal and external quality processes?

As will be seen in the rest of the report, the external drivers have led to quality cultures when the institutions have managed to seize upon changes in the external environment and to embrace quality assurance processes that are fit for purpose and engage the university community.

21


The EQC survey showed that most institutions had the main structures and processes in place to ensure the quality of their activities, particularly for teaching and learning. The data also revealed that for 52% of institutions these developments are very recent: 36% introduced these processes between 2005 and 2009; 16% are currently designing or planning them (Loukkola and Zhang 2010: 21).

The Quality Culture project (EUA 2005 and 2006) noted that the introduction of internal quality assurance processes is a transforming moment in an institution. If done well, it can promote effectively a quality culture. Therefore, it is useful to start with the different ways in which internal quality assurance processes have been introduced and the effects this has had on developing a quality culture. Several contrasting examples, based on the interviews, are provided. This particular discussion ends with an example of changing an internal quality assurance approach. It is included here because introducing change requires the same sensitivity to the goal of strengthening quality culture as when quality assurance processes are introduced for the first time.

4.1 Engaging the academic community

First example

In the first institutional case, the engagement of the university community was felt to be essential in the development process, particularly in the context of a recent merger. In the words of the QA officer:

During the three years preceding the merger, the QA office organised a series of seminars, events and “Quality Days” to discuss quality assurance processes. Some of these events targeted the whole staff; others were more focused toward students, internal auditors, managers, etc. Surprisingly, there was no opposition because the university community knew that the merger was an obligation and that quality assurance processes could be the tools to achieve a better merger.

Part IV. Introducing or changing internal quality assurance processes


PartV:Scope and

organisation



22

The same university ensured engagement at lower levels of the university. Each department and faculty was asked to produce a quality assurance manual that complemented the university’s manual. A faculty representative explained the process:

The faculty decided that the quality assurance manual should not be viewed as an administrative tool but something that would be of benefit to us. The faculty QA coordinator wrote the draft manual and it went through consultation and several iterations. Eighteen months later, it was presented, discussed and adopted at a meeting gathering all students and staff. The faculty decided to introduce QA in a practical way with the goal of ensuring fairness and standardised processes, regarding examinations or grading as examples. We also clarified responsibilities.

These consultations seemed to have resulted in broad ownership of the internal quality assurance processes within the university. None of the rank-and-file academics expressed the least negative comment or any veiled criticism. Most importantly, when asked to describe the approach to internal quality assurance arrangements, there was wide agreement as to their purposes and the spirit behind them.

Second example

A contrasting case is provided by a second institution. The QA officer was appointed by the rector and asked to produce a quality assurance system in response to new requirements set by the national QA agency. Although a vice rector for quality was newly appointed as well, she had other responsibilities. As a result, the QA officer seemed to have worked in isolation and executed this top-down order without much consultation. When asked about the rector’s or the vice rector’s involvement in defining the broad outlines of the QA system or in engaging with the university community, many interviewees responded that they appeared to be disengaged and uninterested.

As reported by various interviewees, the result of the QA officer’s work was very detailed and took no account of disciplinary cultures. A student questionnaire was developed but not piloted. Students felt that some of the questions were irrelevant to them. The reaction from academics was very negative as well. Thus, one interviewee stated:

Today the academics see the quality assurance processes as a burden with which they must comply. The academics do not feel ownership of the concept and feel detached from it in their everyday activities.

Some interviewees designated the national QA agency as responsible for this outcome but the agency’s responsibility was contested by others who pointed instead to the national bureaucratic culture that permeated the university.

Third example

One institution sent out a questionnaire to alumni who had graduated ten years ago as a way of introducing quality assurance processes. The questionnaire results went to the university curricular committee to amend the curricula, develop learning outcomes and an internal evaluation process based on these, and address other issues that arose from the alumni questionnaire. While, in principle, the results of the alumni questionnaire should have raised awareness among academics of the need to introduce changes, the plan backfired because it was driven centrally and did not take as its starting point definitions of academic identities. This was a specialised fine arts institution and although artists are used to receiving feedback on their art products, according to one interviewee, they are not used to receiving feedback on processes and outcomes. “This means that they are not used to asking: What are my objectives when I am teaching and what teaching processes are needed to achieve them?”

23


The academic staff rejected the plan and offered to produce a new plan, which was done respecting academic values and their view of their profession. A committee was formed following an open call for participation to frame the new quality assurance processes. The committee produced a policy that stressed an approach that seeks to embed quality assurance processes in the professionalism of teachers:

In order for the evaluation of courses to achieve the desired goals – namely to improve the quality of teaching – it must be an opportunity for the teachers to reflect on and develop their own teaching skills. It must be clear to the teachers that the evaluation results represent a useful tool for their personal further development as a teacher.

The policy further suggested that the evaluation process must be structured in such a way as to provide information that is important to the teachers and that it should not be used as a controlling instrument.

4.2 Using existing experienceAnother way of introducing quality assurance processes is to build upon existing experience. 32%

of respondents to the EQC survey noted they had used pilots for this purpose (Loukkola and Zhang 2010: 23). Thus, in one of the ten institutional cases, a faculty developed a quality assurance system that was recognised through a national award, which was later scaled up to the whole university. The principle of scaling up the faculty system was not contested, although there were muted criticisms of some aspects of the concept.

In another case, however, a faculty with solid QA experience proposed to scale up its own process to the university, with the result that the plan was rejected. It seemed to have come too soon after an institutional merger, when questions of identity were still paramount.

A successful example was provided by the case of two merged faculties, with differences in quality assurance experience. The newly-created faculty discussed the different approaches now represented and came up with a new approach that combined both experiences. The two faculties did not have the same depth of QA experience and the new approach constituted a small step back for the more experienced one. This step back was accepted in the interest of ensuring broad faculty ownership.

4.3 Introducing quality assurance through staff development

The first round of the Quality Culture project (EUA 2005: 20) noted that “If institutions wish to introduce an internal quality culture, they must start with staff development plans to avoid that quality processes [...] are perceived as a threat to individual career development”. Two universities in the EQC sample followed this lead and introduced their quality assurance through, or associated with, staff development (for more details, cf. 6.5.2).

In one case, a very small group of academics from different faculties was formed by the vice rector, with the addition of a colleague from a nearby institution to serve as a secretary and to provide an external view. The group examined the ESGs and a variety of documents from QA agencies and came up with a


24

rough outline of activities in seven areas (e.g., research-based education) which they considered to be important. This was presented to the top leadership of the university and was followed by a call to academics to design staff development courses for their colleagues.

Similarly, another institution used the metaphor of the “avalanche” to refer to the process by which, in the future, a committee will work with a number of colleagues to develop their capacity to evaluate their teaching. These colleagues will then be asked to work with their own colleagues. The planned workshops will address: (1) the benefits of getting feedback as a way to improve learning; (2) individual teaching concerns and (3) the different ways to elicit student feedback.

4.4 Changing a quality assurance approachQuality culture is fragile and, as mentioned earlier, introducing change appears to be as sensitive

as when introducing quality processes for the first time. One institution was in the process of changing its approach to quality assurance. This university had appointed a senior academic as the first QA officer who had stressed the engagement of the community.

Philip (a fictitious name) played the role of the facilitator of quality culture. He set the tone. He was very patient and allowed people to engage in a dialogue. The quality culture was developed and embedded by Philip in a very soft way.

With the appointment of a new QA officer (also a senior academic), this university changed and strengthened its approach to internal quality assurance. The university used the argument of the current economic crisis to convince the academic community of the necessity for urgent change. The discussion about the change started a year before, with the establishment of a formal committee that conducted a desk research to identify ways in which other universities approach quality assurance. It sent out a staff questionnaire to evaluate current QA practices in the university and conducted focus group discussions (cf. 6.2). This resulted in a report to staff that included both the results of the questionnaire and the desk research.

The involvement of the campus community did not go beyond this. According to the new QA officer, “there is less need for dialogue now because the quality assurance processes are well known and accepted”. The new process was discussed only with the first set of faculties that will be affected. However, the academics who were interviewed, including some who had been directly involved in the new process, had veiled criticisms of the new approach’s managerialism, even if they recognised that some aspects were improved over the previous processes that had ended up generating QA fatigue and work that was somewhat irrelevant.

One specific challenge faced by this university is that the new QA system was introduced in a context of the national fiscal crisis and that, according to one interviewee (and confirmed by others):

This is creating negativity on the ground regarding the new QA processes. The university’s efforts are going beyond the national QA requirements in order to demonstrate to the government that the university is an effective and efficient organisation. There are more structures, more managerialism. Academics understand the current situation and the need to provide a response but they are concerned that this may go too far in changing the humanistic character of the university.

25


4.5 ObservationsTo conclude this section, four observations are in order.

Dissenting or shared values: it is interesting to note that, when discussing the degree of disagreement within the university about definitions and purposes of quality assurance processes, the view from the central administration (whether the QA officer or the vice rector) was not always accurate. Thus, in the first institution where, it will be recalled, all interviewees spoke with one voice and positively about the QA processes, the vice rector noted, however, that “there is a strong, divided opinion about internal quality assurance processes. There are different views among staff and it will take a couple of years to reach wide acceptance”.

While it is possible that the choice of interviewees was biased toward “champions” of the QA processes, in the other two institutions, the QA officer had chosen “opponents” to the QA process for some of the interviews. These were indeed critical but often in a constructive way. This suggests that central administration may be overly sensitive when its decisions are discussed or challenged. This was confirmed in a fourth institution whose vice rector had warned that one of the deans to be interviewed for EQC was critical of the process. It turned out that his criticism was not about the principles or the approach but about an implementation detail that the dean felt was not suitable for the size of his faculty.

Quality assurance or quality enhancement: some universities are more sensitive than others in the vocabulary they use to refer to internal quality assurance processes. Thus, one institution did not use the words “quality assurance” but spoke of quality management and quality enhancement. The word “assurance” carried the risk of making the academics defensive. Rather than talking about how to ensure that quality is good enough, the academics could talk about how to make the quality visible and to improve quality levels. Another university also rejected the words “quality assurance” because the connotation had become negative and evoked control; instead the institution used the words “evaluation” and ”feedback”. Two other universities distinguished between quality assurance (compliance with national requirements) and quality enhancement (institutional initiatives to improve quality).

Externally or internally-defined QA processes: several universities felt that the scope and definition of internal quality assurance arrangements as set by their national QA agency were not appropriate to their purposes or too narrow. These institutions went beyond the national requirements and took the initiative of defining their quality assurance systems in a way that fit their own mission, objectives and values. This involved a degree of risk-taking and demonstrated their institutional self-confidence.

The importance of institutional self-confidence is confirmed by Reichert in her study of factors of institutional diversification in England, France, Norway, Slovakia and Switzerland. She pointed out that there is a tendency for English institutions “to adapt to perceived standards and expectations” set by their quality assurance agency (QAA). Reichert noted that many English institutions tend to play it safe and that, “according to QAA representatives, the institutional behaviour and its willingness to take risks seem to be positively associated with the degree of institutional self-confidence” (Reichert 2009: 33).

Sustaining a quality culture: when redesigning QA processes it is important to acknowledge the weaknesses of the previous processes and the challenges raised by a new context. In other words, ensuring engagement and broader ownership must be an ongoing effort in order to sustain a healthy quality culture. The work is never done.


26

Part V examines the scope of internal quality assurance arrangements, as discussed by interviewees, the distribution of roles across the leadership, students, staff and external stakeholders; the role of curricular committees; the QA office; and the governance of quality assurance. Each aspect is discussed and an overall analysis is presented at the end of Part V, which focuses on elements that are conducive to building a quality culture.

5.1 Scope of internal quality assurance arrangements

It is clear that all universities are interested in the quality of the education on offer. Thus, the interviewees focused on the internal quality assurance arrangements for teaching and learning although the questions put to them were very general. Therefore, the main body of this report will also focus on teaching and learning.

5.1.1 research and service to society

Monitoring research or service to society hardly came up during the interviews, although the EQC survey shows activities in these two areas. Thus, 79.3% of the institutions replied that their institutional quality assurance processes cover research activities and 47.7% that they cover service to society. Interestingly, however, when the responses were crosschecked with other survey questions, a total of 97.3% were reported to have quality assurance activities for research and 95.9% for service to society (Loukkola and Zhang 2010: 19-20).

5.1.2 Administrative services

Generally, the mature QA systems evaluate and include academic support services (e.g. libraries) in internal review processes of departments or faculties (as specified by ESG 1.5) but, again, administrative

Part V. Scope and organisation


PartV:Scope and

organisation


27


services were rarely brought up in the interviews. Some universities, however, have developed or are developing quality assurance processes to deal with the administrative line. Sometimes this falls within the remit of the QA office and sometimes not. Thus, one university created a new post to develop a quality assurance process for administrative services. The person in charge spoke about her mandate in the following terms:

To drive the performance of high quality support service provision, which is characterised by fragmentation and a weak tradition of looking at and measuring performance. The academic culture of independence has leaked into the support services.

Her work rested on one principle: “support services (finance, human resources, counselling, library, advising, etc.) should behave like a service industry; that is, understand what they are trying to achieve, measure it, compare it and improve their performance”. She intended to introduce a performance review of administrative services every three to six months.

According to her, there had been a transition from quality as an add-on to quality as a central theme in the university:

There is now greater acceptance of quality and performance, a concept that is much stressed lately. The key words now are measuring performance. Academic leaders accept this new development, driven by quality purposes and competitiveness. It is a discipline of management and we are trying to build a single, successful organisation. The rector is very vocal about performance. She wants the university to be integrated and customer-oriented and this can only be delivered through discipline.

Her vice rector recognised, however, that there is a great deal of resistance to this new approach among administrative staff. This hard approach to quality assurance was buffered by one of the heads of services who was interviewed and who explained that his staff was defensive and does not see the need for change. He told them that “the goal is not to look at individuals but at processes” and he asked them to think collectively about ways to improve what they do: “This gave them a degree of ownership of the change”.

Although it is difficult to generalise based on the limited evidence, the fact that in this example the responsibility for developing the quality assurance of administrative services was charged to a person (administrative staff) other than the QA officer (academic staff) may have led to a harsher approach.

The resistance of administrative staff was also reported by interviewees from other institutions. Most interviewees mentioned that generally heads of services are more open to quality assurance processes but that rank-and-file administrative staff resist. Interestingly, administrative staff were rarely involved in the development and discussion of quality assurance when it is first introduced (cf. 5.2.3), which may explain their resistance.

5.1.3 Institutional performance

The quality assurance of the institution as a whole emerged as a topic only in three institutions even if the EQC survey found that 55% of respondents used institutional performance indicators to measure progress in achieving strategic goals (Loukkola and Zhang 2010: 28). The three examples discussed in the interviews are as follows (see also Section 6.3):

• In one institution, one vice rector was tasked with the identification of key institutional indicators that would assist in institutional steering. This development was taking place in the context of the recent law that enlarged university autonomy.

• Similarly, in another case, a recent law introduced changes to the governance and the status of the universities. Universities are now public entities, with an external board that appoints the


28

rector. They can invest, buy and sell property. This institution emphasised that it intended to work along the line of New Public Management: streamline the management structure, revise internal structures and the salary scale and devolve responsibility to the appropriate local level. The rector was developing the quality assurance dimension of this proposal.

• In a third case, the university organised an annual management review, which provided important data on a variety of performance indicators (number of students who graduate, number of publications, etc.); how the quality system is working; if there have been any complaints. The senior management team examined the problem areas and made decisions about resources.

It is difficult to draw any conclusions from this very small sample beyond noting that all three institutions were implementing very significant changes and felt the need for institution-wide steering mechanisms during a critical period. These changes came about as a result of national policies that enlarged the scope of institutional autonomy in the first two cases, and of a merger in the third case.

5.2 roles and responsibilitiesOne of the main conclusions of the Quality Culture project (EUA 2005 and 2006) was that a successful

quality culture engages the leadership and the grassroots (i.e., academic and administrative staff and students) as well as external stakeholders. The following sections consider the extent to which this principle is followed in the ten institutions.

5.2.1 Leadership

All ten universities had created a post of vice rector (or equivalent) in charge of quality assurance or quality enhancement). (In one institution, there were three vice rectors involved in various aspects of quality enhancement.) The emergence of the post demonstrates how strategic quality has become. The EQC survey revealed that for 66% of institutions, the senior leadership took the lead in developing the processes and that in those cases where leadership was not leading the process, “it monitors, makes decisions or facilitates the process” (Loukkola and Zhang 2010: 23).

This was certainly the case in the ten institutions: the initial steer came from the top but the continuous involvement of leadership varied from almost total disengagement (one university) to some degree of engagement (all other universities).

Several institutions stressed that responsibilities were devolved to the lowest possible level and that the senior management team was involved only in case of serious problems. This approach is illustrated by the following quote from a vice rector who emphasised the importance of combining bottom-up responsibilities and top-down steer:

Leadership role is crucial. If the leadership does not believe in the importance of quality assurance, it will not penetrate. The rector communicates the importance of this by talking to the deans. It is then up to them to discuss it in their faculties. One of the most important aspects of quality assurance arrangements is that they offer a tool for the leadership: through the measurements (input, output and process), they can get an overall picture of how the university is doing.

The vice rector noted that the deans were now full-time and appointed: “These changes brought more clarity of decision-making while maintaining shared governance. It also increased the dynamic capacity of the institution”.

29


The changing status of deans, while not yet typical in continental Europe, points nevertheless to an emerging trend: that of integrating deans in the top management teams rather than confining them to merely representing their faculties (Sursock 2010). If this trend grows, it will strengthen institutional strategies and institutional accountability and, by implication, will lead to an effective use of quality assurance results. Indeed, an effective quality culture and quality assurance mechanisms can only be developed if (i) responsibilities are shared and accountability lines are clear and (ii) if the leadership – at all levels of the university – is pulling in the same direction and able to persuade the staff that they are important players (and beneficiaries) in achieving the strategic orientation of the institution.

5.2.2 Students

The EQC survey showed that students’ involvement in internal quality assurance varies (Loukkola and Zhang 2011: 24). It is difficult, however, to see exactly what this finding means based on the survey results only. The interviews provide further details and clarification.

In eight of the ten universities, students were involved in internal quality assurance processes and discussions either because there is a culture of student involvement at all levels of the university and a clear recognition of their added-value or because it is a national requirement. One student leader explained student involvement in her university as follows:

Students are involved at all levels of the university, from planning courses to university strategy and budgets. They do not receive formal training but there is an introductory meeting. I was very surprised to see that the university was very open to student input in decision-making. Students are consulted in a meaningful way but they are more useful on some topics (courses) than on others (university budget). At the faculty or department level, students are involved in course planning and course development. They sit on the department councils which give input to the faculty. Teachers get together in programme groups (no student) and in programme boards that review programme proposals (with students).

This student, however, added that “students are not always treated as adults and they don’t always act as adults”. This was seen as a causal relationship that could be changed: giving students more responsibilities would increase the chance of changing their behaviour.

Other interviewees reported problems with student participation:

• In one institution, students were seen as making sometimes unreasonable demands. This was confirmed to some extent (although not acknowledged) by the head of the student union who stated: “Students are invited to many meetings to discuss issues and there are lots of opportunities to be active but sometimes the university does not listen to what we say”.

• Some interviewees remarked that the student population is changing and that students have become more demanding. Sometimes, but not always, this is linked to student fees. One interviewee noted that students are more aware of their rights: “Academics have become compliant with whatever they are asked to do because they know that the students know about the ombudsman’s office and the process of lodging a complaint”.

• In another institution, students were involved in the process of designing the internal quality assurance processes, but they dropped out of the committee when they realised that it was focused on teachers’ development.

• Finally, one institution recognised that “the involvement of students in quality assurance processes is one of our weaknesses. Most do not know the system and are concentrating on their studies. We should improve communication with the students”. One person suggested that this might be due to the institutional merger that increased staff workloads; the two student unions, too, had to


30

merge and this also created difficulties. The interviews showed, however, that depending on the size of faculties (small) and their cultures (non-hierarchical, team-oriented) students were involved in very meaningful ways at other levels of that university.

5.2.3 Staff

In the sample of universities involved in the study, academic staff members were in charge of the QA office, when there is one. Sometimes their academic background has faded somewhat because of their long-term involvement in the quality assurance unit but they had started out as academics. In addition, academic staff members are part of curricular committees (cf. 5.3) and are the subjects of the internal evaluations through the student questionnaires, the requirement to submit an annual report of activities or other means.

Administrative staff members generally occupy junior posts in the QA office unless they have gained expertise and the capacity to relate to academic staff. There seems to be a trend to include them in the internal quality monitoring, usually starting with the learning support services.

While academic staff members were usually consulted in developing internal quality assurance systems this was not always the case with administrative staff.

5.2.4 External stakeholders

The role of external stakeholders seems to overlap with their involvement in governance structures, curricular boards and other bodies (cf. 5.3). This confirms the findings of the EQC Part I report that concluded that “external stakeholders are more likely to be involved in preparing a curriculum [...] the results indicate that their role is usually that of information providers rather than decision makers” (Loukkola and Zhang 2010: 25).

5.3 Curricular committeesIn many universities, curricular committees gather representatives from these different constituencies:

staff, students and external stakeholders.

The EQC survey revealed that 85% of respondents indicated that their institution has programme committees (Loukkola and Zhang 2010: 29-30). This high score is probably the result of the Bologna reforms. The introduction of modularisation, the stress on individual learning paths and learning outcomes implies the necessity to lodge the responsibility of curricula in pedagogical teams rather than with individual teachers.

These committees are an important building block of the internal quality assurance process even if they are not recognised explicitly as such. Thus, the link of the programme committees to the internal quality assurance system varied. Three examples are presented below. These examples are from universities that have recently introduced quality assurance arrangements. The more “QA-mature” institutions tie the internal quality assurance processes and the programme committees more closely. Instead of having two separate quality and programme committees, the latter is asked to consider the evaluation results when reviewing or redesigning programmes and to provide input in designing quality assurance processes.

31


First example

In the first institution, all programmes were checked systematically and rigorously. Each curriculum went through a departmental, faculty and university approval process. The programme committees, at each level, monitored the description of courses and made sure that they included the appropriate information. They also ensured that the description was clear to the students and that the content matched the intended learning outcomes. The participation of students in these committees was viewed as essential: they provided a quality check, particularly by shedding light on whether syllabi were comprehensible to their peers.

The heads of the faculty programme committees were convened by a vice rector every two weeks. These meetings also included the quality officer and representatives from the academic support services (library, computer). These meetings were not focused on the work they do locally but on cross-university issues, including quality assurance. The quality assurance plan was on the table all the time and was used as a reference but – interestingly – the faculty programme committees did not receive the evaluation results. Consequently, it is only towards the end of an interview that the head of a programme committee recognised that his committee was an intrinsic part of the quality assurance arrangements. During the first two-thirds of the conversation, he described his role as totally unrelated to quality assurance processes.

In the near future, the faculty programme committees will be responsible for carrying out programme reviews on a three-year cycle based on set criteria, such as student involvement, expected and achieved learning outcomes, articulation of research and teaching, gender and ethnicity.

Second example

Another institution was in the process of establishing programme committees, which will include academics, one to two students, one to two external stakeholders, an administrative member of staff and a programme director. The faculty will be responsible for approving new programmes, ensuring the link between teaching, research, innovation, international developments (e.g., finding partners for joint programmes and signing the agreements) and ensuring that the programmes align with the university strategy. The programme application will need to include information about the programme, why it is needed, who else in the university or in the country offers such a programme or elements of it, the resources required, etc. When applications are approved by the faculty, they will then go to a university programme committee.

The intention was to give these committees clear quality assurance responsibility in addition to their task of approving study programmes. The committees will collect, analyse and discuss student feedback and propose recommendations for improvement. Student feedback will become the organisational responsibility of the programme director who will decide the format (e.g., discussion or in written form). The director will give the results to the faculty and the faculty will send it to the university. If a course is poorly evaluated, the programme director will be in charge of discussing this with the teacher and making recommendations. The programme directors’ salaries will be tied to their performance in carrying out these duties.

Third Example

In a third university, each programme had a programme director but with no QA responsibility. In the words of one interviewee: “the programme director organises; the faculty quality committee evaluates”. Nevertheless, the programme director sat on the faculty quality committee which helped articulate the two bodies. This faculty quality committee, which included a student and an external stakeholder, reviewed the evaluation results of the previous year and sent a report to the faculty council. There was also a university quality committee, chaired by the vice rector for quality, that gathered the deans and the faculty directors of quality.


32

5.4 QA officesThe EQC survey revealed that 62% of institutions had a central QA office with specialised staff.

Eight of the ten universities in the interview sample had a QA office, thus representing a higher percentage than in EQC Part I. The ninth university had an academic centre within the rectorate and the tenth was considering whether to create a QA office. In addition, a number of universities in the interview sample had QA coordinators in faculties or departments.

All QA offices were located centrally and reported directly or indirectly (i.e., via the vice rector for quality) to the senior management team. Interestingly, two students from a single institution identified the independence of the QA office from the faculties as one of the success factors for their university’s quality assurance initiative. Both mentioned that the office reports directly to the rector, thus preventing lobbying from the faculties.

The QA office usually serves several functions. The following identifies what the QA officers highlighted as the primary function of the office:

• Supportive role and providing expertise: the QA officer visits every faculty and every department regularly and is invited by them to provide expertise in developing their quality assurance processes. As one QA officer put it:

The visits to departments and faculties made the difference. People felt listened to and were happy that someone from the central administration came to them. You can’t imagine the amount of coffee I drank, the number of biscuits I ate and the stories I heard!

• Coordination role: particularly when the process is devolved to faculties or if there is a process of evaluations that is organised by the university. In one example, the QA officer created a network of the faculty and departmental QA officers. He supported their work via email and convened them two to three times a year. In another example, the central QA unit supported the university quality committee, communicated with the faculty QA committees and coordinated their work.

This coordination role can be extended to other units in the university when QA is seen as linked to two other functions: staff development and institutional data collection/analysis (cf. 6.5 and 6.3). As one QA officer stated:

My main interactions are with the vice rector for quality, the head of pedagogical development with whom I brainstorm regularly and a senior planning officer, with whom I work on issues connected with institutional data performance.

• Interpretative role: one of the tasks of the QA officer is to interpret the national and European quality assurance requirements so as to adapt them to the institutional context. When one academic was asked about national QA requirements, he stated:

I do not know them and I do not want to know them. We have decided to avoid the question altogether. It is the QA officer’s job to follow national developments. Our aim is to develop a quality culture adapted to the institution.

Thus, the successful QA officers ensure that these national and European requirements are embedded in, and owned by, the university.

• Monitoring role: The office provides instructions, collects information, flags problems, but does not get involved in solving them.

33


• Administrative role: organising and preparing external evaluation visits or processing institutional questionnaires. Thus, one of the QA officers described a typical day as: preparing or processing the results of the evaluation questionnaires and – at the moment of the interview – speaking to vendors to select software to process the questionnaires. Interaction with the campus community was done primarily by email.

Based on the interviews, it is clear that the effective QA officers were those who combined successfully all these roles and managed to share the responsibility for quality assurance across the university. It is a multifaceted position that requires a complex set of social skills and personal attributes.

5.5 governance of quality assuranceThe EQC survey showed that 40.5% of respondents had a quality committee at some or all levels of the

university – central/faculty/department (Loukkola and Zhang 2010: 20). This was confirmed by the interviews. Almost all universities in the sample had a process to analyse the results, formulate recommendations and discuss them across the various committees (departments, faculties, university, including the governing boards) but this did not mean that they were able to engage various, pertinent committees in the quality assurance work. Thus, in one case the institutional research office (responsible for collecting institutional data) reported to one vice rector and the rector but not to the whole senior management team (cf. 6.3.1). In other cases, as mentioned earlier, the link between the quality assurance processes and the curricular committees was unclear (cf. 5.3).

The quality assurance technical literature emphasises the importance of closing the feedback loops and it is clear in the EQC Part I report that these feedback loops are not functioning properly and need attention. The following three examples illustrate clear sequences in closing these loops.

First example

In one university, the dean met with the heads of departments monthly and one meeting a year was devoted to quality issues. It was at this level that the majority of issues arising from the internal evaluations were addressed. The deans’ council, chaired by the rector, discussed quality issues twice a year. In this institution, responsibilities were devolved to the lowest possible level and the reporting lines were very clear. The decentralised management of quality assurance allowed local adaptations and local responsibilities.

Second example

In the second example, departments gathered and analysed a range of evaluation data and identified areas of concern. This analysis fed into the faculty’s yearly appraisal and went to the faculty QA subcommittee, which included the heads of departments, the persons at faculty level supporting learning innovation, students and the faculty QA officer. A report was then sent to the university QA committee, which included representatives from various levels of the university, from the senior management team to faculties and departments and one student. The university committee had two subcommittees that each included three students. The first looked at teaching/assessment and was chaired by the university QA officer; the second looked at the learning experience and was chaired by the vice rector. The two subcommittees considered a range of data (results of the student questionnaires, course performance data, external examiners, etc.) and developed an annual response that went to the plenary committee. It was then sent to the academic council (the widest body in the university) and to the university board.


34

Third example

The results of the quality assurance process in the third institution were given to the vice deans, the deans, the rector and the senate. Each faculty produced a development plan with goals and key indicators, measured annually. The senate was informed yearly of the results. The university community was convened every two years to discuss progress. The results of the student questionnaires were sent to the faculty quality committee, which prepared a summary report that was sent to the heads of departments and went to the faculty council. The summary report was anonymous and included general averages and an analysis of trends. The faculty council, which discussed QA as a regular item on its agenda, made recommendations and ensured the follow up. The recommendations were given to the heads of departments who were expected to report back on progress. The summary reports and the recommendations were posted on the faculty web page and were sent to the QA office and the rector’s office.

5.6 ObservationsThe examples provided in Part V suggest several observations:

Student engagement: if the departments and faculties manage to create a home for the students successfully, this will increase their engagement. In addition, it was noted that student engagement usually grows as they progress during their university years and become increasingly interested in being consulted. Furthermore, by and large, interviewees recognised that the successful involvement of students is conditional upon there being an effective student union – one that is encouraged and supported by the university to gather and disseminate information to students. In some cases, institutions or faculties provided students with opportunities for leadership development, thus ensuring effective participation.

The QA officer as a cultural mediator: It is clear from the interviews that, beyond the priorities of the QA officers and the functions of the office, the more successful quality officers are those who have ready access to the senior leadership, the social skills to communicate effectively with and to engage and support academics. Playing the cultural mediator between the top and the grassroots seems to be an effective way to grow and, most importantly, to sustain a quality culture, by ensuring that there is shared understanding of its purposes. It also ensures that the QA officers feel like an integral part of the academic community and accept, as one QA officer put it, “that academics like to argue, contest and criticise; they must be allowed to do so and be persuaded that they are not losing their academic freedom through the quality assurance processes”.

Furthermore, when the QA officers have other functions or are involved in a broader set of issues than simply quality assurance processes, their work benefits. It allows them to keep their fingers on the academic pulse and to be perceived as more than the (potentially controlling) QA officer but as someone who knows the institution well and is an active contributor to its development and that of the staff. Thus, in at least two institutions, the QA officers were involved in other activities. One sat on various committees that put him in contact with the rector and the head of academic staff development: e.g., teaching advisory committee; a steering committee to improve the masters’ programme offer. Another one had additional functions (manager of the academic development unit, e-learning, study skills and access support) and was part of the senior management team.

Coordinating QA, staff development and institutional data collection/analysis: it is important that these three functions are well-coordinated. Staff development schemes ensure that internal quality assurance processes are useful because they provide academics with assistance in improving their teaching or introducing innovative pedagogies, rather than just penalising them for failures. The collection and analysis of key institutional data support institutional planning and management and quality assurance processes. With QA, these functions are crucial in allowing institutions to monitor effectively areas of

35


strengths and weaknesses and to develop appropriate, coordinated actions in response to gaps in provision. These functions require qualified staff members who work together to support institutional and personnel development.

Feedback loops and bureaucracy vs. streamlined structures and clear responsibilities: too much reporting and too many committees may give the impression that the feedback loops have been closed. Although the multiplication of committees serves to distribute responsibility and is, in principle, a good way to ensure that a quality culture develops and is not simply bolted on, these committees are not a “silver bullet” to the extent that they do not necessarily lead to the development of a quality culture. As one interviewee stated bluntly and pointedly, “committees are breeding like rabbits”, thus signalling the need for some moderation. It is important to realise that “committology” and paperwork may lead to bureaucracy. Interestingly, in the EQC sample, the university that had the simplest way of closing the feedback loops was also the most effective in grounding quality assurance processes because it defined clearly the responsibilities while closing the feedback loop. Therefore, it is crucial to identify who needs to know what and to distinguish between what is necessary vs. what would be nice to know.


36

Part VI discusses processes and tools related to quality assurance. These include formal and informal student feedback, alumni consultation, key performance indicators, and internally-organised reviews. Because the quality of an institution is directly related to the quality of its staff, Part VI discusses also staff management and development.

Institutions are putting in place a range of instruments to monitor quality. Before discussing these instruments in detail some general remarks are in order. There seems to be several factors affecting the choice of QA instruments and the general approach:

• In some cases, there are national requirements that are set by the ministry, the QA agency or professional accreditation bodies. The EQC survey revealed that this was true for nearly 52% of respondents (Loukkola and Zhang 2010: 23).

• In a minority of cases, universities started with ready-made packages such as ISO or EQFM; the EQC survey found this to be the case for less than 9.5% of respondents (Loukkola and Zhang 2010: 28). These are sometimes then further adapted or have evolved over time while maintaining some elements of the initial approach.

• Finally, based on the interviews, the disciplinary background of the person in charge of developing the QA approach seemed to loom large in how the processes are constructed: engineers, mathematicians, or statisticians tend to favour quantitative approaches and instruments. Other disciplines tend to favour qualitative and perhaps less systematised approaches.

An additional factor that influences the design of quality assurance arrangements is the tendency to imitate the arrangements of nearby universities. Given the methodology of this study, it was impossible to test this hypothesis but research done in Portugal as an example showed clear patterns. Portuguese institutions tend to favour centralised systems in which the rectorate or the presidency defines the QA procedures or systems and delivers them to faculties. In addition, about ten years ago some public polytechnics started to implement certification systems, mainly ISO-9001, and a wave of imitation followed across the country, led successfully by specialised consultancy firms, and resulting in the generalisation of ISO in this particular subsector. Some of the polytechnics did not keep their ISO certification but still maintain the main procedures after they adapted them to their own institution. ISO-9001, CAF, EFQM and other systems are also well spread among private institutions. In Portuguese public universities, these systems are usually only implemented in the administration or in specialised faculties such as medical schools, chemistry departments, and business schools (Fonseca 2011).

Part VI. Tools and processes


PartV:Scope and

organisation


37


6.1 Collecting feedback through questionnaires

6.1.1 Student questionnaires

Student questionnaires are the most common way for institutions to introduce quality assurance processes. Thus, 71.6% of respondents to the EQC survey used student questionnaires and 92.4% of those take the results into account in the assessment of teaching staff (Loukkola and Zhang 2010: 27).

For some institutions, as Fonseca (2011) pointed out for Portugal, student questionnaires “are the quality assurance system”. Therefore, it is troubling to note the range of problems connected to this popular instrument. This is the area where – to be blunt – worst practices are found.

Thus, interviewees reported such problems as: questions that are not meaningful to the students or are ambiguous; a single questionnaire that is used across the university, with no possibility of adapting a section locally in order to capture some specific teaching practices (e.g., clinical training); questionnaires that do not seem to have been tested through pilots; resistance from the central administration to revise a questionnaire despite widespread criticism; or questionnaires that are too long. As one interviewee stated, “poor questionnaires encourage teachers to teach poorly in order to get good scores on the questionnaires”.

In addition, the results of the questionnaire and their use are somewhat opaque to the students.

The EQC survey notes that, of the institutions administering student questionnaires, 58.5% state that students are informed of the outcomes and the resulting actions taken; the percentage drops to 6.3% when it comes to making the information on teachers’ aptitudes and performance publicly available (Loukkola and Zhang 2010: 27). In most cases, the results go to the teachers and their heads of departments but, naturally, the conversations that follow an evaluation are private. Students do not know that such conversations are held and in which cases and some teachers do not discuss the results of the questionnaire with the students. Even when the results are posted, the students do not always know that they are available.

The interviews also showed that not all academics were aware of whether faculty deans, vice rectors or even rectors received this information. This knowledge seemed to depend on the interviewee’s status. It was only generally available if the university had clearly identified the process and informed everyone.

To complete this picture, many institutions have moved to on-line questionnaires. The move to on-line questionnaires has generally resulted in a lower return rate and no solution was reported to be found. In one institution teachers were asked to take students as a group to the computer centre to fill in the questionnaire; and even this resulted in a relatively modest 40% return rate. According to one interviewee, there have been critical voices in his country about the on-line questionnaires:

They have cost a great deal and there is no evidence that they are useful. Teachers have ignored them and some universities have withdrawn them because they promote a compliance culture.

Thus, survey questionnaires must be thought through carefully, particularly if they are used to make decisions about teachers. The EQC survey found that of the 61 institutions that have the ability to dismiss academics, 49 take the survey “results into account in the assessment of teaching staff” (Loukkola and Zhang 2010: 27). To the extent that student questionnaires are used and can affect academic careers, their quality is paramount and much attention needs to be paid to this instrument. To address these concerns, some institutions or faculties have developed other ways to handle the questionnaires.


38

First example

One institution developed an electronic system that requires teachers to give feedback to the students. The questionnaire has three university-level questions. A faculty or a department may add some questions, which are the same for all feedback collected by that faculty or department. Teachers can then add their own questions. The university has limited the questionnaire to 12 questions. The results are collected and analysed by the teacher who must respond to the students with feedback, and explain what s/he will do to change or why s/he does not think that a student’s recommendation is worth pursuing.

Second example

Another institution launched its questionnaire by asking all the faculties to bring the questionnaires that each was using to the table. Each faculty was happy with its questionnaire but felt that the other questionnaires were really poor. The vice rector in charge thought that all the questionnaires were poor; he asked an expert in questionnaires (who was working in the university) to develop a set of questions, which was piloted by volunteer teachers. Students were part of this pilot group and were actually the instigators of this new initiative since they felt that the existing faculty questionnaires did not lead to any results.

Today, the university has two questionnaires: one that is formative and voluntary, that teachers are encouraged to give three weeks into the autumn term; and one that is summative and is required by law at the end of each course. Teachers can choose which questions to ask from a set of 25 questions; they are encouraged not to ask more than five to ten questions and to vary their questions in order to keep students interested. They are also encouraged to use the formative questionnaire and to make immediate changes, if needed, in order to encourage students to respond to the summative questionnaire. The questionnaires, the results and follow-up are posted online. The open comments which sometimes deal with personal issues regarding a teacher are not public.

The focus of the mandatory questionnaire is on the course, the learning outcomes, the strategy for reaching the learning outcomes, the support the students got from class activities, from readings and other assignments, and from their teachers. Most importantly, the students are asked how much time they have spent on a specific course as opposed to other courses. This allows the administration to judge if the course workload is adequate in relation to the credit units assigned.

6.1.2 Alumni questionnaires and consultation

The EQC survey shows that alumni are rarely involved in formal quality assurance processes and that nearly 12% of institutions have no contact with their alumni. On all items tested, the rate of response concerning alumni contacts is lower than for any other groups, including external stakeholders. (Loukkola and Zhang 2010: 24):

• 41.4% of institutions elicit information from alumni in an informal way;

• nearly 35% of institutions ask them to respond to regular surveys; although the response to two other EQC questions showed that 40.5% track graduate employment and 45.5% provide information on alumni employment by study programme (Loukkola and Zhang 2010: 26);

• nearly 26% involve them in self-evaluation or other evaluation activities;

• 19.4% of institutions involve them in consultation bodies;

• nearly 10% of institutions involve them in governance bodies.

39


These data were reflected in the interviews. Few interviewees brought up the alumni’s involvement in quality assurance processes but there were a couple of interesting initiatives at the faculty level. Thus, one faculty dean reported that the alumni were gathered as frequently as needed, about four times a year.

A faculty in another institution instituted a very successful annual gathering of alumni. About 1 000 alumni come, sometimes with their families. The success of this event prompted other faculties in this institution to follow suit. These gatherings were reported to be a very effective way of getting informal feedback on trends in industry and business and improving the curriculum.

One institution administered an alumni questionnaire with 20 questions that are set nationally and 30 set by the institution. According to one interviewee, the return rate was poor because it was a new initiative in a country not used to polling its people and where there was a general sense that one’s opinion does not count. The high number of questions may also be an issue.

Alumni relationship is probably under development in many other institutions; therefore, the link with quality assurance is not always made. It is still early days and it is difficult to draw any conclusions at this point.

6.2 Collecting oral and informal feedback Perhaps the most effective way of grasping how well students are doing is to talk to them. Thus, one

respondent to the EQC survey noted:

Much of the quality is dependent on the informal nature of staff/student relationships. The increasing calibration of quality indicators has led to a concern that this relationship will become formalised and thus less productive (Loukkola and Zhang 2010: 16).

There were many examples of fostering oral feedback in the EQC interview sample as the following examples – mostly taken at faculty level – illustrate.

First example

For instance, one faculty, which was forced to shut down for a few years, reopened about 20 years ago convinced that it would not survive if its teaching was not constantly improved. As a result

• right after every examination period, the students meet with the teachers to discuss the semester. This is done before the teachers mark the exams, which implies a high degree of trust

• after the students receive their marks, three or four teachers get together with the students to discuss the examination questions and to explain the marks. One of the students indicated that about 90% of students attend these meetings

• each teacher gets the relevant results of the student questionnaires and summarises main strengths and the weaknesses that he/she plans to address. This is posted online. Every new semester, the teachers give the results of the questionnaires to the previous cohort and inform the new cohort of the changes they brought to the course in response to the students’ evaluations.


40

Second example

In another example, the dean of a large faculty (6 500 students) explained the following feedback mechanisms. All students give feedback by using an online questionnaire that was developed centrally and focuses on specific study programmes. This feedback is complemented by a number of discussions with students in the faculty. The dean and vice deans collect students’ comments throughout the year by meeting quarterly with the head of student representatives to get feedback. In addition students are divided into groups by levels and programmes and they elect heads of groups who meet twice a year with the dean and vice deans. Every two years, students are asked for an overall evaluation of programmes.

Third example

Yet another faculty organises three feedback meetings a year to which all academics and staff are invited. The meetings focus on a specific theme but generally the student union extends the agenda by voicing any student concerns. In addition, teachers have an open door policy and consult students informally, and the students fill in an on-line questionnaire.

Fourth example

One university organises feedback in an unusual way: in addition to the standard questionnaire, an external teacher will meet with a group of students, at the end of a module or the beginning of a semester, to discuss what they liked and did not like and their recommendations. The actual teacher is not present during these conversations but will get a summary of the discussion.

Fifth example

One university collected oral feedback on its evaluation procedures through focus groups (cf. 4.4). These are an effective instrument for evaluating services or testing new ideas. Focus groups usually gather six to ten people with a similar profile (e.g., students, academic staff, and administrative staff). The facilitator develops five to six questions for sessions that last from 60 to 90 minutes. To the extent that these sessions are essentially group interviews, many of the same guidelines for conducting interviews apply to them.

6.3 Collecting feedback through institutional data and key performance indicators

6.3.1 Institutional data

The EQC survey results showed that nearly 99% of institutions had data information systems and that for 93% this was a centralised system. The collected data include generally: student progression and success rates (87.7%), the profile of the student population (83.2%), the teacher/student ratio (65.5%), the results of the student questionnaires (53.6%), the cost of learning resources (44.1%) and tracking graduates’ employment (40.5%) (Loukkola and Zhang 2010: 26).

41


Having such information available, however, does not mean that there is an institutional research office and the EQC survey is silent on this point. It is possible that data collection is performed by several different units depending upon the sort of information collected. The interviews were not greatly revealing either except for one institution that has had such a unit for a very long time and volunteered information about its activities, which included:

• Tracking graduates of all programmes and levels through annual surveys to analyse how long it took them to find a job and at which level, their salaries, and their satisfaction with their job and education. These data were analysed to produce a list of jobs per subject/level to help current students think about their professional future. They also allowed the university to examine the fitness of its educational offer for graduate employability.

• Annual student evaluations by study programme and level

• Annual data collection of all study programmes on the basis of a set of indicators

• Analysis of social and financial data such as student grants and housing

• A survey of entering students to identify those at risk of failing: family situation, high school record, etc.

In addition to its periodic data collection, this institutional research office was capable of responding to any statistical data collection needed. Despite its capacity and the skills of its staff, however, the university was reported as not having exploited fully the institutional research office in the past. This is clearly changing now as the university is gearing up to introduce internal quality assurance arrangements. It is important to note that this is occurring at the same time as the institution is implementing a range of national policies that have increased institutional autonomy and introduced competitive funding incentives. Thus, the head of the research office was identified early on as one person who should be interviewed to highlight an area that had become central to the university.

6.3.2 Key performance indicators

The EQC survey revealed that 55% of institutions “had defined a set of key performance indicators to analyse their contribution to the achievement of institutional strategic goals” (Loukkola and Zhang 2010: 28). The data include such items as: retention, pass rates, graduation rates, diversity of student profiles, internationalisation, etc. Typically, such data are assembled at faculty level and sent up to a central committee, which provides a response in the form of corrective actions. These seem, in general, to be actions to be taken across the university. This central steer can sometimes be adapted at faculty level. Thus, faculty deans, in consultation with staff, can define the faculty’s priorities to align them with the university’s priorities.

Among interviewees, only in one institution were there no complaints about how the key performance indicators were interpreted perhaps because it lodged responsibility for the interpretation of results and the response, first and foremost, at the faculty level:

The annual review provides important data on a variety of performance indicators (number of students who graduated, number of publications, etc.); how the quality assurance system is working; if there have been any complaints. The faculty executive team examines the problem areas and makes decisions about resources.

All other interviewees mentioned problems of interpretation once the results reached the central level. Two typical responses, from two institutions, are provided below:

Since it is a centralised QA system, the vice rector is responsible for quality assurance. She receives the data and reports, analyses these and produces a general report. The problem is with the interpretation of data: she is too far removed to know the contextual details.


422 For a description of good practices in organising internal reviews, cf. Irish Universities Quality Board (IUQB) (forthcoming).

There is a sense that the metrics are insufficient to capture the complexity of the reality on the ground (e.g., pass rates) and are open to a variety of interpretations. Interpretation from the university centre tends to be inadequate and to simplify the situation.

In some cases, the metrics themselves were criticised: “Science and technology disciplines seem to be more accepting because they are used to the metrics but the humanities are particularly resentful because the metrics used are not adequate (they publish books rather than articles)”. One department was reported to be currently gathering international benchmarks to show that the university’s benchmark for the number of PhDs per supervisor was not adapted to its disciplinary field.

Finally, one university was in the process of introducing centrally-defined key performance indicators: “The focus on benchmarking has been resisted by academics. They view such a process as a sideshow to their work and a burden”.

6.4 Internally-organised reviews and articulation with external reviews

The EQC survey showed that institutions “seem to conduct a variety of processes in a variety of combinations” to monitor and update their study programmes and that 57% of respondents evaluate them regularly (Loukkola and Zhang 2010: 30) but the survey data are unclear as to the processes in place. The interviews provided further evidence. Alongside programme committees that examined study programmes (cf. 5.3) and the reviews by external agencies (e.g., national or international quality assurance agencies or professional accreditation bodies), some universities use external examiners (e.g., Ireland and UK) or conduct unit reviews.

6.4.1 External examiners

In the case of one university in the EQC sample, the faculty QA officer coordinates the faculty assessment officers and is responsible for the cross-faculty scrutiny of marking and grades. Faculty assessment officers are members of subject teams who keep track of assessment issues. They ensure that courses and assessments are fit for purpose, collect all marks, check the assessment spread sheets and prepare the documentation for the university examination board. In effect, they are the internal examiners. The external examiners look at a sample of examinations from across the mark range. They look at each module and at the mark profile of the students to identify inconsistencies.

6.4.2 Unit reviews

Two universities organise an internal review process at the level of schools (i.e., a structure combining several faculties) or subjects2.

• In one institution, a panel of external and internal members review schools or subjects every four to five years. Their role is advisory and is based on a self-assessment report. The review report goes to the school, the faculty and the university. One interviewee reported:

Colleagues find this process very useful. It is an opportunity to reflect on areas of strengths and weaknesses. It is collegial. Everyone understands that the panel is acting in an advisory role and

43


the process is constructive. This is true in my school but varies across schools. The level of trust depends on how the school and the panel are led in the process. My head of school presents this as an opportunity rather than a threat. I was part of a recent internal review of another school where the panel understood its role as advisory but not the school; staff were on the defensive.

• The second institution proceeds in much the same way. This university has recently changed the focus of its internal reviews, moving them to the school level. The previous system produced evaluation fatigue because it was focused on smaller units and those academics who were based in several units were constantly being reviewed. The university conducted focus groups to discuss its internal review process. Some of the key concepts that emerged to characterise a good review included: efficient, streamlined, meaningful, sharply focused, fit for purpose, transparent, user-friendly, linked into university strategy.

The university hopes that the new process will result in less evaluation fatigue. One academic who participated as a reviewer in this new process noted that:

The process is now more focused, more streamlined and based on defined criteria such as level of performance or student satisfaction. In the past both the self-evaluation and the review reports were very long. Now, thanks to a new data system, schools produce data in tabular forms. The analytical text is just 15 pages (instead of 60). This is very useful and the background can be understood during the review meetings. The data are generated by school administrators who are part of the self-evaluation group. It does take time to collect them and the school that was recently reviewed noted that this work plus everything else that is being asked of them (operational and strategic plans, etc) are quite a bit of work. But during the oral report session, the school came around to the process and found it to be very useful because of the panel’s constructive comments. In the past, the recommendations were generally confined to asking for more financial and human resources.

In addition, the university now asks all schools to produce a plan with targets, to review this plan and present it to the university leadership every six months in order to embed a continuous improvement process. According to the QA officer:

In the old system most academics saw quality assurance as a nuisance that happened every five years. Now, they will need to make it a central, daily concern focused on quality improvement. In this way, the quality culture will grow, in an operational way, rather than diminish.

6.4.3 Coordinating internal and external QA reviews

Accreditation by professional regulatory bodies provides another external driver to improve quality levels but they pose a specific challenge to the universities that have a formal internal review process. To avoid being constantly reviewed, one school planned the internal review and the accreditation visits to coincide so that the two panels visited the school at the same time. From this example and that of other faculties that have to undergo professional accreditation, it is clear that coordinating the internal with the external review processes is a challenge. It can result in more work and in having to deal with two sets of requirements and standards.

It is important to note that both institutions discussed above (cf. 6.4.2) were located in countries where the quality assurance agency conducted institutional audits. Therefore, the topic of programme evaluation/accreditation conducted by national agencies did not come up during the interviews. The experience these two institutions have had in coordinating both an internal and external process is worth pondering particularly because the vast majority of national agencies across Europe work at the programme level (ENQA 2008) and that some of the ten institutions interviewed are in the planning stage of developing internal reviews. The development of internally organised unit reviews will probably result in negotiating a shift to institutional audits with the external agencies in order to lessen the QA burden and contain expenses.


44

6.5 managing academic staff

6.5.1 recruitment and promotion

All universities have had processes for recruitment and promotion but these have not always been explicitly connected to QA processes. The EQC survey report stressed the link between managing academic staff and the maturity of QA processes:

The results of the survey lead us to believe that the QA of teaching staff may be closely connected to the timeframe of the introduction of QA. While almost two thirds (63.4%) of all institutions surveyed have specified their own requirements for hiring teaching staff, 76.8% of those who had introduced their QA system before 2000 have done so (Loukkola and Zhang 2010: 27-28).

Evidence from the interviews reveals a slightly more complex picture. Two broad categories of institutions can be distinguished in relation to their capacity of managing staff.

Nationally-set requirements

In the first category, institutions have limited autonomy in this regard and requirements for recruitment, promotion and dismissal are set nationally. The interviews revealed that although the national requirements for managing academic careers may vary, they seem to result in some frustration among academics particularly, as in the case of one institution, when great stress has been placed on innovative teaching, which has been defined nationally as development of blended learning. One academic noted that “the theoretical courses do not easily lend themselves to a different format than lectures and that the problem is that quality is linked to innovation but innovation should not be imposed on all courses”.

In another case, however, despite stringent national requirements, the sense of institutional pride was strong and seemed to overshadow any existing frustration. The national requirement is that all academics must be reassessed every five years. This is a competition that is open to externals. Candidates are examined holistically for their research, teaching material and university service. There are threshold requirements for research activities which are evaluated on the basis of standards developed by the science academy. Candidates who fail the process are usually re-appointed for one year and told what they need to do to improve their chances. Academics gain tenure after they have been reappointed three times. Academics are evaluated and their salary is based on their performance. They can be fired if there is cause. The academics who were interviewed seemed to feel a sense of pride in their institution and to accept this process because “it puts pressure on all academics to improve continuously their teaching and update their knowledge”.

Institutionally-set requirements

In the second category, we find institutions that are free to set their own criteria for recruitment, promotion and dismissal. Interviewees from these institutions have stressed three conditions for an effective approach to staff management and quality assurance.

• Whether quality is at the heart of the institutional strategy and teaching a clear institutional priority: thus one institution placed high value on teaching as a core part of the university and had institutionalised processes to reflect this priority. By and large, however, many academic staff

45


reported that they feel multiple and contradictory pressures to perform well in research, teaching, community outreach and university service (see also Hénard 2010). Even if there are teaching awards and salary increases based on teaching merit, research was felt to be more important for promotion in many institutions. Most worrisome, many academics in a few countries (most particularly in Ireland) reported increased workloads and the need to work around the clock, including on weekends. In Ireland, each academic must submit yearly objectives and must undergo a yearly evaluation. One academic staff pointedly stated: “I feel like a salesperson now. Education has become a commodity rather than an experience”.

• Whether the institution has the flexibility to offer different types of contracts and promotion pathways depending on where academics want to focus their priority at different points during their careers: e.g., teaching, research or innovation. This was clearly the case in one institution.

• Whether there is leadership capacity within the institution. This was brought up by only one senior interviewee but was discussed at a recent EUA conference which concluded that ensuring good staff management requires: “(i) providing leadership and development programmes for senior leaders to allow them to mentor and manage their staff effectively and (ii) building a community of purpose that includes both academic and administrative staff” (Sursock 2010).

While it is true that these institutions also have had quality assurance processes in place for a long time, thus confirming the correlation established by the EQC report, they were also the ones with the largest scope of autonomy within the sample. This confirms an early finding of the Quality Culture project, which established a correlation between autonomy and internal quality assurance processes (EUA 2005). In other words, the longer the institution has enjoyed autonomy, the more likely it is to have a clear strategy, a sharp institutional profile and more developed staff management processes based on clear internal quality assurance arrangements.

6.5.2 Staff development

The EUA Quality Culture project (2005 and 2006) had identified staff development as an important building block of internal quality assurance and quality enhancement. Staff development can be developed by the human resource unit and by a centre for teaching and learning. The EQC survey revealed, however, that only 38.3% of respondents had “a unit responsible for staff development” and 47.7% had “a unit responsible for pedagogical innovation (or equivalent) that offers support to teachers in developing teaching methods” (Loukkola and Zhang 2010: 20). About 30.6% had both types of units.

In six of the EQC universities, interviewees spoke of staff development schemes to improve teaching skills and promote innovative pedagogies. These schemes were linked – implicitly or explicitly – to the internal quality assurance arrangements. The examples below can be seen on a continuum: from the most implicit to the more explicit links between the two areas.

Implicit l ink to quality assurance arrangements

Three universities offered staff development but without linking it specifically to the internal quality assurance arrangements. Interestingly, the “maturity” of QA processes was not a factor in determining whether such a link existed. Thus, the first university had introduced its quality assurance system very recently while the other two were the most “mature” in the EQC sample.

The first university was adapting to a new legal framework that stresses the need to develop innovative teaching, which is defined as using new information technologies. The staff development courses were limited to providing information-technology support for blended-learning initiatives. Staff development courses are optional and usually taken up by the younger academics in order to ensure the national accreditation (habilitation) of their qualifications.


46

The other two universities had a much more developed academic development offer through a centre for teaching and learning and other activities. In one institution, this centre has the following functions:

• to provide academic development courses, including a post-graduate certificate and a master’s degree. It offers training workshops and various events, including an annual conference. These are focused on such issues as teaching methods, designing curricula, etc.;

• to develop various policies related to teaching and learning and develop a teaching and learning strategy;

• to support the development of the virtual learning environment, including providing equipment in classrooms;

• to enhance the civic engagement of students through volunteer work in the community and internationally and to embed it in courses through service learning;

• to conduct research on curriculum development and higher education policies.

According to one interviewee, these somewhat disparate elements had the value of putting the centre in close contact with academics. This positioning – unlinked to QA – gave it more freedom and academics were more favourably disposed toward the centre as opposed to the QA unit. The language the centre uses was also academic rather than managerial.

In addition to this centre, the university has introduced a mentoring scheme to support career development, which appears to have been well received. Academics will be encouraged to self-evaluate and identify their mentoring needs. There is a plethora of well-regarded training offered by the human resource department and academics also have access to training to understand how to fill in their time sheet (for full costing) and prepare their performance appraisal process.

The third example concerns one university with the longest record (among the EQC sample) in having internal quality assurance arrangements in place. It also has a centre for teaching and learning to support teachers. The unit offers a compulsory credit-rated course for new staff; e-learning support for staff; one-on-one coaching; and coaching of course teams interested in changing their teaching approach.

In addition, the university has faculty coordinators who are responsible for promoting new ways of teaching and disseminating good practices. Information flows from the centre for teaching and learning to the faculties and vice versa.

Towards a more explicit l ink to quality assurance arrangements

Staff development may not necessarily be linked to the internal quality assurance arrangements at university level although these links may be more apparent at the departmental and faculty level – if somewhat informally. Thus, the central human resource department in one university offers courses every week to academic staff leaders to develop their managerial and leadership capacity: e.g., leadership and change management, corporate responsibility, effective management, etc. These courses are either developed by in-house academics or by external people and are not linked to quality assurance arrangements. They are voluntary and open to anyone who is interested.

At the departmental level, teachers are asked to go and observe more experienced teachers in their classrooms and to discuss with colleagues new ways of teaching. In addition, if the dean identifies a promising academic with poor teaching skills, this person could be sent to take a course offered by the university or to visit another institution abroad. A new and interesting initiative was launched by a dean, which consists of pairing an international guest professor and a local junior teacher as his/her assistant in order to develop better teaching locally.

47


Explicit l inks to quality assurance arrangements

As mentioned earlier (cf. 4.3) two universities decided to introduce quality assurance arrangements and staff development as a joint scheme.

In the first university, the whole quality assurance scheme was based upon and introduced as staff development, albeit without referring to it as a quality assurance initiative. The approach was qualitative and included:

• seminars bringing together academics from all faculties to discuss such activities as designing a course, identifying learning outcomes and developing the relevant examinations;

• talks where experts speak about specific topics: e.g., implementing ECTS;

• training academics, based on a national requirement for a 10-week course: The first step was to issue a call for academics who might be interested in leading this training. These volunteers developed several courses:

- An introductory course that is focused on discussions about the “idea of a university” (i.e., the ideals, academic values), how to develop student-centred learning, etc.

- Three 5-ECTS courses on a variety of topics, such as student assessment. As a result of this specific course, a few academics were selected to form a group to work further on this topic.

- Other thematic courses include using information technology in education, coaching, combining theory and practice, teaching in English in a multicultural environment.

At the end of the course work, all academics have to undertake a capstone project to show how their teaching has developed. They are encouraged to publish this as an article or to make a conference presentation. After this, they are encouraged to take courses every other year in order to consolidate and continue their professional development.

Finally, the university funds special projects. The call for proposals goes out at the end of the academic year for the funding of ten projects (5 000 Euro each) for collaborative work in quality development. These projects become the topic of seminars once they are completed. Students are involved in these efforts.

In the last example, a university, also in the early stages of introducing internal quality assurance processes, has reshaped the staff development unit and ensured that it works closely with the quality assurance unit. An academic working group produced two papers that defined aspects of quality for pedagogy, curriculum and organisational aspects (e.g., classrooms, coordination) as well as the development of teachers. As a result, an academic development offer was developed and included didactics, conflict resolution, etc. Other elements to be developed include enabling teachers to develop qualitative evaluation procedures by themselves and to document that they have done it; the possibility for teachers to invite a colleague for class observation and feedback; and to organise an external peer review.


48

3 For instance, a teacher could stop class a few minutes early and ask students to respond briefly – orally or in writing – to a couple of questions, e.g.: What was the most important thing you learned during this class so far? What important question remains unanswered? What classroom activities or assignments have been most effective in helping you learn this semester, and why? What classroom activities or assignments have been least effective in helping you learn this semester, and why?

6.6 ObservationsStudent questionnaires suffer from a number of failings that require some attention. Four principles

seem to be particularly important. The questionnaires must:

• ask questions that are relevant for identifying effective teaching, without pushing particular pedagogical agendas or creating biases. A form that asks whether the instructor used blended learning, for example, implies that all good teachers must do it, but this is not necessarily appropriate for all courses;

• have a limited number of questions and allow some local variations;

• ask questions of students about their engagement in the learning process in order to convey to them that good teaching and learning is the result of an active partnership between the teacher and the learner;

• provide feedback to the students on what has been changed on the basis of the questionnaire results.

Mid-term evaluations, conducted by teachers and providing feedback to the students rather than the administration is probably the best way to ensure that improvements are implemented and, most importantly, that students see the results3.

It is essential not to depend on the questionnaires alone because their reliability seems to consist of identifying only the best or the worst teachers. In addition, research has shown that the questionnaire results are not generally analysed to control such factors as class size and type and that it is important to do so. In the USA for instance, where such research has been conducted, large classes or theory courses usually receive low ratings (Glenn 2007).

Recent studies have also shown that “student evaluation scores are so weakly related to learning that it is a serious error for colleges to use those evaluations as the primary tool for judging teachers” (Glenn 2009). Therefore, when student questionnaires are used they should be only one part of a package of instruments for evaluating teaching and learning – a package that could include e.g., analyses of written instructional materials, particularly course descriptions (cf. 5.3). The results of these evaluations should be analysed and pedagogical teams should come up with action plans. In parallel to these activities it is essential to develop training and advisory services to teachers who are interested in improving their teaching skills (cf. 6.5.2).

Informal and oral feedback seems to be an effective feedback mechanism provided several conditions are met:

• This kind of activity is best left to the departments or faculties in order to ensure that groups are kept relatively small.

• The department or faculty has encouraged trusting, open relationships of staff with students and created a sense of home for the students.

• There is an understanding that different students will feel comfortable giving feedback in different ways. Therefore, best practice seems to be a combination of approaches (written and oral; formal and informal) as the examples provided above illustrate.

49


• The involvement of student representatives is essential: they can model appropriate behaviour and voice students’ concerns in an anonymous way.

Key Performance Indicators can be contentious; the arguments concentrate on the division of responsibility between the central leadership vs. the faculties/departments and individual academics. Interestingly, the debate regarding key performance indicators mirrors the debate that surrounds external quality assurance. Many observers have noted that external quality assurance is about power to the extent that it requires defining quality and the processes by which quality levels are measured. Thus, the questions are: Who defines quality? For what purposes? These two questions can be asked about internal quality processes as well.

Should definitions and measures of quality be defined centrally, by the grassroots or both? Ultimately, key performance indicators have to make sense to the grassroots while supporting institutional strategies. This suggests a sequence of actions: (i) developing a strategy and then (ii) identifying key performance indicators aligned with it. Both steps must engage the university community in order to ensure ownership. (For specific examples of key performance indicators, cf. CPU 2010; IUQB 2008).

This also suggests the need for good leadership and for building a real community of purposes and shared values. In fact, one of the vice rectors interviewed mentioned that she had proposed to develop a quality charter for her university. Her rector instructed her to produce first a charter laying out the key values of the university which would serve to underpin the introduction of formal quality assurance processes. This being said, it is an open question whether any university will, or indeed should, achieve perfect harmony. A certain degree of debate and disagreement should be expected, particularly in an academic community.

Investing in staff: The Quality Culture project (EUA 2005) identified staff development as a weak area in most universities in Europe. Since then, it seems that there has been growing awareness of the need to develop academic support services and staff development in general as illustrated by the range of interesting initiatives that have been presented above. This confirms the positive correlation established by the EQC Part I report between the “maturity” of the QA arrangements and the existence of academic development units (Loukkola and Zhang 2010: 20), and suggests that this will be an area of further growth as institutional QA arrangements mature and continued stress is placed on the quality of educational provision and active learning.

From the evidence provided in the interviews, the link between quality assurance processes and staff development need not be a concern: it can be explicit or implicit. The choice probably depends on several factors, including how quality assurance arrangements are perceived and the institutional emphasis on teaching and learning. Thus, it could be surmised that a positive quality culture and a priority on good teaching would “cope” well with an explicit link to staff development. If there is “QA fatigue”, however, some institutions take great pains to dissociate the two by distinguishing between quality assurance – i.e., responding to external requirements – and quality enhancement – i.e., responding to institutional and individual developmental needs.

Regardless of the choice made, it is important that staff development is made available and fully developed and that those responsible for it have the required expertise. Often, this expertise is available internally. Drawing on it would ensure that staff development is congruent with the organisational culture and institutional objectives. This could be complemented with external expertise, which would serve to show that other institutions are also active in this area.


50

There are five key conclusions to be drawn from the discussion in Parts IV, V and VI:

First, it is important not to rely on a single instrument, such as the student questionnaires, particularly if it shapes staffing decisions. There must be a mix of several instruments to ensure good intelligence. In addition, it should be borne in mind that these should not be neutral; they must be related to institutional strategies and – ultimately – to academic values. Their costs and benefits must be reviewed regularly: this includes not only financial costs and benefits but also psychological aspects (e.g., do they lead to unnecessary stress or unreasonable workloads?) and whether they really contribute to embedding an effective and shared quality culture, supporting the institutional strategy and providing accountability toward students and the wider public.

Second, the most effective internal QA arrangements are those that derive from effective internal decision-making processes and structures. Having clear accountability lines and clarifying responsibilities at all levels ensure that the quality assurance system is kept as simple as possible while closing all the feedback loops and this should, if anything, reduce bureaucracy by limiting data collection, reports and committees to what is absolutely necessary. It is crucial to identify who needs to know what and, furthermore, to distinguish between what is necessary vs. what would be nice to know. In addition, as has been noted several times, students and staff feel at home, first and foremost, in their faculties and departments. This argues in favour of an optimal balance between the need for a strong institutional core and a degree of faculty responsibilities, between the need for an institution-wide QA approach and some local variations in faculties.

Third, leadership is essential to give the initial steer and the broad frameworks of quality assurance mechanisms. Leadership should facilitate internal debate – and even tolerate dissent – in order to make sure that quality assurance processes do not end up being imposed and simply bolted on. Linked to this, the type of language used by the leadership and the QA officers in describing the QA arrangements cannot be dismissed as trivial. The more academic and the less managerial it is, the more likely it will make inroads in the institution.

Fourth, it is essential to invest in people through staff development, to avoid internal quality assurance arrangements becoming punitive. It is encouraging to note the pace at which staff development schemes are growing but centres for teaching and learning, staffed by professionals, are still a rarity. This will require attention in the years ahead particularly because of the renewed emphasis on student-centred learning in the Bologna Process.

Fifth, both institutional autonomy and self-confidence are key factors in the capacity of institutions to define quality and the purposes of their internal quality assurance processes and to ensure that these are in line with their specific profiles, strategies and organisational cultures. In doing so, these institutions are sometimes confronted with their external quality assurance agencies’ processes, which might be mismatched. It is essential that the internal and external processes are viewed together and that the higher education community – the institutions and the agencies – negotiate the articulation between the two sets of processes in order to ensure true accountability and avoid duplication of evaluations and thus QA fatigue.

Part VII. Conclusions – structures, cultures and history

51


Furthermore, the report alluded to the fact that, like external quality assurance, internal processes are also about power and that internal quality assurance can be contested if it does not successfully engage the university community. In fact, both internal and external QA processes provide a window into deeper undercurrents of political cultures. Institutions are part of a specific cultural environment, even if there are variations in organisational cultures across a single institution or across institutions located within a single cultural environment. Therefore, it seems essential to end with broader issues of political cultures.

7.1 Quality assurance is about power and ideology

Many observers and researchers have noted that quality assurance is not a neutral concept, but closely related to questions of ideology and power. It is about who defines quality and in which ways (e.g., Stensaker and Harvey 2010). Harvey and Williams (2010: 7) note “that analysis of quality should not be detached from purpose and context and that quality has political dimensions and is about more than (consumer) satisfaction”.

In this context, Amaral and Maassen note that the balance between improvement and accountability depends on who is responsible for external quality assurance. They point out that external quality assurance was initially a concern of academics but that it:

has become progressively a matter of public concern in the 1980s and 1990s with an emphasis on quality improvement and accountability. The balance between these potentially conflicting objectives shifts towards improvement when academics have a strong voice, and towards accountability when the will of the governments predominates (Amaral and Maassen 2007, xii).

If this is true of external quality arrangements, what about internal quality arrangements? Are they more likely to focus on improvement rather than on accountability, since it is the academics who are defining them? This was not found to be necessarily the case. All ten universities had academics behind the development of quality processes. Their involvement, however, was not enough to define the main approach as oriented towards improvement because a wider set of factors is at play when the purposes of quality arrangements are being defined.

Towards the end of each interview, a question about the main purposes of the quality arrangements was put. The replies from the ten institutions could be placed on a continuum: from improvement to accountability. Four examples illustrate this continuum.

First example

In the first example, the university seemed to have generated a culture of compliance because the processes were too complex and detailed and left no room for local adaptation in faculties and departments. In addition, the lack of institutional autonomy in respect of staff management meant that improvement was a challenge. As one dean put it:

The main purpose of the quality system is to have a quality system! Improvement is difficult because of the system’s excessive complexity and because the deans and the university in general have no levers to improve the quality of teachers. If a problem is identified, I can only persuade the teacher to change.


52

In other words, it is a problem of institutional autonomy since staff members are civil servants.

Second example

The second university managed to promote a culture of continuous improvement as the following quotes from several interviews illustrate:

The general philosophy is focussed on embedding quality in the daily work processes in order to support the mission and vision of the university and to improve the quality of activities.

The quality arrangements make the university’s work more concrete, more visible and show how units link to the university process and their position within the institution. It demonstrates the way that students can affect and improve the teaching process.

The purposes of QA are to have common practices and to ensure that activities are carried on at the same quality level. Through the quality work, we can identify best practices and this improves quality levels.

Third and fourth examples

Two universities combined both improvement and accountability by distinguishing two different sets of processes – quality assurance and quality enhancement:

Quality assurance is about day-to-day monitoring of standards for reporting purposes; it is about administrative processes of quality control. There is a psychological resistance to quality assurance and distinguishing it from enhancement allows us to focus on improving the student learning experience.

The remainder of Part VII explores possible reasons for these different approaches by examining the complex interplay of a range of external and internal contextual factors.

7.2 Participation and bureaucracyIt is difficult to separate institutional definitions of quality cultures from several dimensions of the

national culture, particularly those that have to do with definitions of participation, democracy and the bureaucratic culture.

7.2.1 Participation

Perceptions of participation are linked to definitions of democracy and affect how new quality processes are introduced in an institution. Based on the interviews, aspects that seem relevant are: the importance given to teamwork and consensus, the ability to negotiate compromises, and the degree of consultation that is felt to be optimal. Thus, one interviewee noted: “We are used to democratic decision-making and people want to be involved in shaping the goals, aims and standards of the university”. By comparison, several interviewees located in another country noted that the lack of experience with opinion polls and a historical sense that individual opinion does not count mean that it is difficult to engage in internal quality processes. As a result, a debate was organised at all levels of the first institution in developing internal arrangements. This ensured that all points of view were heard and taken into account in the search

53


for an acceptable compromise. In the second institution, a recognised expert in QA systems developed the internal arrangements, which acquired legitimacy based on this expertise.

Linked to democratic and participative cultures, the level of trust in the higher education system is also important and this is partially dependent on whether an open higher education market exists, (i.e., if all types of providers are allowed to operate without prior approval and are regulated through accreditation). As Karl Dittrich4 suggested (personal communication), where the level of trust is high, external QA agencies view their role as confirming a well-functioning system. In countries where higher education institutions are distrusted, the sector has to submit to a controlling and intrusive quality assurance culture on the part of the QA agency. A high level of distrust is partly linked to the existence of shady providers or diploma mills. Any uncovering of such a provider casts a long shadow across the sector and leads governments to require controlling and punitive external QA mechanisms regardless.

7.2.2 Bureaucracy

Even without such shady providers, however, QA agencies can show distrust toward the higher education sector and be intrusive in how they frame their requirements. The role played by the QA agency and its determination of the level of detail required by internal quality processes can be linked to perceptions of bureaucracies, particularly since nearly 52% of institutions responding to the EQC survey stated that their quality concept was based on requirements set by their national QA agency (Loukkola and Zhang 2010: 22). Thus, universities are not immune to a managerial approach to quality if the bureaucratic national culture leads them there. Extreme cases in the interview sample were represented in two countries where interviewees characterised the national (and therefore the institutional) bureaucratic culture as focused on paperwork, and as being inefficient and very hierarchical.

Max Weber, the sociologist who pioneered the study of bureaucracy, describes it as a progressive step that ensures rational decision-making but he also spoke of the “iron cage” of bureaucracies (loosely translated by Spencer from the German stahlhartes Gehäuse) when they go too far in rationalising processes (1994: xvi). Robert Michels spoke of the “iron law of oligarchy” (1915) which can affect even the most democratic organisations. In their most extreme forms, bureaucracies produce oligarchies and morph into top-down organisations. They limit human freedom and potential and become segmented by too much specialisation, resulting in a loss of the sense of community and belonging. Thus, one interviewee stated: “There are now pedagogical teams but professors are individualistic, defensive and conservative; they do not want to change and coordination efforts within faculties are difficult to put in place”.

Because bureaucracies are socially constructed, it is also civil society that can deconstruct them, as often argued by Jürgen Habermas who speaks of the need to have public debates in which all opinions are valued equally. If the grassroots in universities can be considered as the equivalent of civil society, then deconstruction of bureaucracy certainly occurred in one of the ten universities, as was discussed in Part IV. It will be recalled that this university had introduced internal quality arrangements in a top-down, centralised manner:

The first steps that were taken in the university were technocratic: it posted a mandatory on-line questionnaire (with mandatory publication of results). Colleagues ignored it or were resistant. They argued that the metrics, the purposes and the use of results were unclear and could not see how it would help them improve their teaching. Resistance was very tough.

Resistance is not always that open. Thus an interesting study of the implementation of performance-based funding (PBF) in German medical faculties showed that – contrary to expectations – managers did not gain ground over academics because senior faculty members oversaw the process and that PBF had to be approved by the university governance bodies, which “gave traditional professional-collegial procedures ample opportunities to intervene” (Schulz 2011).

4 Chair, Nederlands-Vlaamse Accreditatieorganisatie (NVAO), the accreditation organisation of the Netherlands and Flanders.


545 Head of Section Higher Education Reform in Germany and Europe, German Rectors’ Conference (HRK), Germany.

Dill (2011) concludes an overview of new public policies in Hong Kong, the Netherlands and Finland with the normative statement that:

... the available evidence supports the view that as universities become increasingly autonomous, the public interest will be best protected by strengthening the collegial processes by which universities themselves maintain their academic standards, validate their research, and assure the value of their service to society.

But this raises the question of how to ensure that a strong collegial process does not lead to the maintenance of the status quo. Thus, a HRK project revealed that the different leadership levels in a university may successfully block one another’s initiatives by “playing by the book” and ignoring what can safely be ignored. According to Barbara Michalk5 (personal communication), there need not be a clash of cultures or an open conflict and everybody carries on “business as usual”. This suggests the need for open discussions and dialogue and effective leadership to change minds and institutional cultures.

7.3 Perceptions of academic rolesParticularly, it is essential to convince academics that assuring the quality of their teaching and of

their students’ learning experience can contribute to their personal and professional development. This issue was explored in all ten universities and interviewees were asked if resistance to quality processes among their colleagues fell into any clear sociological patterns:

• Many mentioned disciplinary cultures: in general, academics in the sciences and the regulated professions were frequently mentioned as being more open to internal quality assurance than their colleagues in the arts, humanities, law or social sciences.

• Others spoke about a hierarchical culture that implies that colleagues feel that students are not in a position to assess teaching quality and about more egalitarian cultures that welcome student input. This dimension is sensitive to disciplinary research cultures: those that tend to work in groups also tend to be less hierarchical.

• Still others mentioned age, but there were no clear patterns across the ten universities: in some cases, older staff was resistant, while in other cases, they were the champions of the quality assurance cause.

• Some mentioned the level of internationalisation of a department: the more international it was, the more likely were staff members to be open to quality assurance processes. International students and staff challenge accepted ways of carrying out activities and provoke a debate about quality levels.

• If promotion and rewards are focused on research performance, the perception of professional role

and academic identities becomes firmly linked to disciplinary expertise rather than teaching.

55


7.4 Institutional contexts: structures and history

The third set of factors that affects the effectiveness of quality cultures is related to institutional contexts, particularly to the degree of centralisation and devolution of responsibility in the university, as well as to the history and the evolution of its internal quality arrangements. These elements can also be related to perceptions of democracy and bureaucracy.

7.4.1 Structures

Thus, to recall two contrasting examples that were presented in Part IV, the introduction and definition of quality arrangements were defined centrally in one institution; this caused acceptance problems in some of the faculties. In another institution, the system was defined centrally but flexibly – each faculty and department supplemented the main system with its own guidelines – thus ensuring broad ownership.

It is important to note that the main factor here is not so much the extent to which the university is decentralised but rather whether:

• responsibilities are devolved and accountability lines are clear;

• there are mechanisms by which faculties are exchanging experience and information;

• the quality system – even if it is defined centrally – allows a degree of local adaptation and therefore broad ownership.

When these three conditions are met, there is a closer interrelationship between quality culture as beliefs and values on the one hand and structures and management on the other.

Situations where decisions on curricular content and pedagogical approaches were lodged at departmental levels but QA processes were defined at central level produced tensions and contested definitions of quality.

7.4.2 history

In addition, as internal QA systems mature, they seem to go through several phases. In the first phase, quality arrangements are implicit and could be called the “Bourgeois Gentilhomme” approach to quality. These institutions “do” quality without naming it or recognising as such6.

Several examples of implicit quality assurance are provided in this report as well as in other studies. Thus, the Trends 2010 (Sursock and Smidt 2010) report noted that changes to the Doctoral level have led to more attention being paid by institutions to quality issues but that these are not explicitly identified as such. Nevertheless, a number of initiatives, all concerned with improving quality of doctoral education, have been implemented: the introduction of new supervision models including professional development

6 The reason for choosing the title of Molière’s play (1670) to refer to implicit systems of quality has nothing to do with the main character’s foolishness – Monsieur Jourdain, a nouveau riche who wanted to pass for an aristocrat. Rather it is for the play’s most famous line, which refers to something that someone does without knowing that it has a name. Thus, Mr. Jourdain takes a philosophy lesson, during which he discovers that he had been speaking “prose” all his life: “By my faith! For more than forty years I have been speaking prose without knowing anything about it, and I am much obliged to you for having taught me that” (Act 2, Scene 4).


567 One institution had introduced ISO because the QA manager had little QA experience initially. There is an attempt to correct

this course now and to develop a more academically-grounded quality approach.

for supervisors; the development of internal regulations and codes of practice as well as agreements signed between the Doctoral candidate, the supervisor and the institution; improvements in standards of access, recruitment and selection; ensuring high standards of the process of the thesis defence; regular monitoring of each Doctoral candidate’s progress and for tracking Doctoral graduates.

Phase I of the EQC project came to the same conclusions in respect of research and service to society. As mentioned earlier (cf. 5.1.1), 79.3% of the institutions replied that their institutional quality assurance processes cover research activities and 47.7% that they cover service to society. Crosschecking these answers with those provided to other questions revealed that a total of 97.3% reported quality assurance activities for research and 95.9% for service to society (Loukkola and Zhang 2010: 19-20).

In these implicit internal quality assurance environments, responsibilities are generally distributed; they are embedded in the definition of professional roles and promoted through staff development. One of the ten universities, for instance, introduced internal quality through staff developments but called these changes “Bologna” (cf. 4.3). Interestingly, the national QA agency could not recognise the existence of these implicit quality processes. This resulted in an external sanction and the judgement that these processes were related to Bologna and not to quality assurance, perhaps because responsibilities overlap and could lead to a view that they are fragmented and that institutional responsibility is unclear. Thus, one of the universities, at an early stage in the development of its quality assurance processes, recognised that when the responsibility is widely shared, the challenge is “to elaborate effective links between individual responsibility on the one hand and institutional responsibility (monitoring, planning, supporting) on the other”.

As QA systems mature, they become professionalised. In one institution, quality was initially defined by the disciplines. The definition has moved recently to the central level, in a context where grassroots experience of quality assurance processes was strong. The shift to the central level increased the managerial dimension and a uniform approach to quality assurance.

The system can then drift toward managerialism. Because it went too far in that direction, one institution attempted to redress this by introducing the concept of “consultancy” in the internally-organised quality reviews. This institution also distinguished between quality assurance and quality improvement.

7.5 Concluding remarksThus, the response to the question as to whether academic or managerial values predominate in

internal quality arrangements is linked in part to their history. As QA systems are introduced, mature and change, they initially – albeit not always7 – stress academic values, go through a phase of bureaucracy and managerialism and, in the best case scenario, go back to academic values in an attempt to have more effective internal quality cultures that can be embraced by the academic community.

In addition, the factors that promote effective quality cultures are that:

• the university is located in an “open” environment that is not overly regulated and enjoys a high level of public trust

• the university is self-confident and does not limit itself to definitions of quality processes set by its national QA agency

• the institutional culture stresses democracy and debate and values the voice of students and staff equally

57


8 The author is grateful to Caty Duykaerts, Director of the Agence pour l’évaluation de la qualité de l’enseignement supérieur (AEQES), French Community of Belgium, for drawing attention to this paper.

• the definition of academic professional roles stresses good teaching rather than only academic expertise and research strength

• quality assurance processes are grounded in academic values while giving due attention to the necessary administrative processes.

Furthermore, an empirical study of how to develop effective safety processes in industry holds some lessons for universities8. Daniellou et al (2009: 103) argue that the degree of engagement of both management and staff determines whether a safety culture exists:

• when engagement of both management and staff is weak, the approach is ineffective and fatalistic (Type A);

• when safety is implicit and embedded in professional roles and implication of management is weak, there is a certain degree of security but no security culture (Type B);

• when management involvement is high and staff involvement is low, the safety process is managerial (Type C);

• it is only when both management and staff involvement is high that one can speak of a genuine safety culture (Type D).

Figure 1: Cultural types

Source: Facteurs humains et organisationnels de la sécurité industrielle : un état de l’art. Les Cahiers de la sécurité industrielle 2009

These four types can be applied to quality culture as well. The three rounds of the EUA Quality Culture project concluded by emphasising that:

Success factors for effectively embedding a quality culture include the capacity of the institutional leadership to provide room for a grass-roots approach to quality (wide consultation and discussion) and to avoid the risk of over-bureaucratisation (EUA 2006: 32).

(-) Management involvement (+)

Professional culture

Fatalistic culture

Integrated culture

Managerial culture

Type B (+/-)

Type A (-/-)

Type D (+/+)

Type C (-/+)

( -)

Staf

f inv

olve

men

t ( +

)


58

These conclusions are confirmed by the EQC study. One of the main goals of the EQC interviews was to ascertain the extent to which (1) the purposes of internal quality processes were agreed or contested and (2) whether there is a quality ethos in the institutions or if quality arrangements are simply bolted on. As described in this report, much has been achieved in ensuring and enhancing quality in institutions, but more remains to be done to make quality culture a reality. This requires the joint responsibility of institutional management and the university community as well as optimising the link between internal and external quality assurance.

59


references

Amaral, A. and P. Maassen (2007) Preface, Quality Assurance in Higher Education, D. F. Westerheijden, B. Stensaker and M. J. Rosa (eds.), pp xi-xii. Dordrecht (NL): Springer.

Conférence des présidents d’université (CPU) (2010) Guide méthodologique pour l’élaboration du tableau de bord stratégique du président d’université et de l’équipe présidentielle, Paris: CPU, http://media.enseignementsup-recherche.gouv.fr/file/2010/68/3/GUIDE_TB_Pdt_Avril_2010_144683.pdf

Daniellou, F., M. Simard and I. Boissières (2009) Facteurs humains et organisationnels de la sécurité industrielle: un état de l’art, Les Cahiers de la sécurité industrielle. France: Institut pour une culture de la sécurité industrielle, http://www.icsi-eu.org/

Dill, D. (2011) Public policy design and university reform: insights into academic change, Plenary paper presented at the Third International RESUP Conference: Reforming Higher Education and Research, 27-29 January 2011, http://www.resup.u-bordeaux2.fr/manifestations/colloque%20international%20Paris%202011/Colloque%20RESUP%202011%20Vol%201.pdf

ENQA (2005) European Standards and Guidelines for Quality Assurance in the European Higher Education Area. Helsinki: ENQA, http://www.eqar.eu/fileadmin/documents/e4/050221_ENQA_report.pdf

ENQA (2008) Quality Procedures in the European Higher Education Area and Beyond – Second ENQA Survey. Helsinki: ENQA, http://www.enqa.eu/pubs.lasso

European University Association (EUA) (2005) Developing an Internal Quality Culture in European Universities: Report on the Three Rounds of the Quality Culture Project 2002-2003, Brussels: EUA, http://www.eua.be/eua/jsp/en/upload/QC1_full.1111487662479.pdf

European University Association (EUA) (2006) Quality Culture in European Universities: A Bottom-up Approach. Report on the Three Rounds of the Quality Culture Project 2002-2006, Brussels: EUA, http://www.eua.be/publications

European University Association (EUA) (2009) Improving Quality, Enhancing Creativity: Change Processes in European Higher Education Institutions, Brussels: EUA, http://www.eua.be/publications

Fonseca, M (2011) Higher Education Accreditation in Portugal: Year Zero 2010, Paper presented at the Third International RESUP Conference: Reforming Higher Education and Research, Paris, France 27-29 January 2011.

Glenn, D. (2007) Method of using student evaluations to assess professors is flawed but fixable, 2 Scholars Say, Chronicle of Higher Education, 29 May 2007.

Glenn, D. (2009) One measure of a professor: students’ grades in later courses, Chronicle of Higher Education, 9 January 2011.

Harvey, L. and J. Williams (2010) Fifteen years of quality in higher education, Quality in Higher Education, Volume 16(1), pp. 3-36, April 2010.

Hénard, F. (2010) Learning Our Lesson: Review of Quality Teaching in Higher Education, Paris: OECD.


60

Irish Universities Quality Board (IUQB) (2008) Good Practice for Institutional Research in Irish Higher Education, Dublin: IUQB, http://www.iuqb.ie/info/good_practice_guides.aspx?article=2c7067a0-e116-4214-953f-d78ebb3f9725

Irish Universities Quality Board (IUQB) (forthcoming) Good Practice for the Approval, Monitoring and Periodic Review of Programmes and Awards in Irish Universities, Dublin: IUQB, http://www.iuqb.ie/info/good_practice_guides.aspx

Loukkola, T. and T. Zhang (2010) Examining Quality Culture: Part I – Quality Assurance Processes in Higher Education Institutions , Brussels: EUA, http://www.eua.be/Libraries/Publications/Examining_Quality_Culture_Part_1.sflb.ashx

Michels, R. (1915) Political Parties: A Sociological Study of the Oligarchical Tendencies of Modern Democracy. New York: The Free Press.

Molière (1670) The Middle Class Gentleman (Le bourgeois gentilhomme), Project Gutenberg, http://www.gutenberg.org/catalog/world/readfile?fk_files=1450815&pageno=18

Reichert, S. (2009) Institutional Diversity in European Higher Education: Tensions and Challenges for Policy Makers and Institutional Leaders. Brussels: EUA, http://www.eua.be/Publications.aspx

Schulz, P. (2011) The conflict between “new” and “old” governance following the introduction of Performance based funding in German medical departments, Paper presented at the Third International RESUP Conference: Reforming Higher Education and Research, 27-29 January 2011, Paris, France, http://www.resup.u-bordeaux2.fr/manifestations/colloque%20international%20Paris%202011/Colloque%20RESUP%202011%20Vol%202.pdf

Sursock, A. and H. Smidt (2010) Trends 2010: A Decade of Change in European Higher Education, Brussels: EUA, http://www.eua.be/Libraries/Publications/Trends_2010.sflb.ashx

Sursock, A. (2010) Conference report, Diversities and Commonalities: the Changing Face of Europe’s Universities, EUA Annual Conference, Hosted by the University of Palermo, 20-23 October 2010, http://www.eua.be/Libraries/EUA_Annual_Meeting_Palermo_2010/Palermo_WGsReport_Sursock.sflb.ashx

Stensaker, B. and L. Harvey (eds.) (2010) Accountability in Higher Education: Global Perspectives on Trust and Power, New York and London: Routledge.

Vettori, O. and M. Lueger (2011) No short cuts in quality assurance – Theses from a sense-making perspective, Building Bridges: Making Sense of Quality Assurance in European, National and Institutional Contexts, Blättler, A. et al. (eds.) Brussels: EUA, http://www.eua.be/Libraries/Publications/Building_bridges_Making_sense_of_quality_assurance_in_European_national_and_institutional_context.sflb.ashx

Weber, M. (1994) Political Writings, P. Lassman (ed.), Cambridge: Cambridge University Press.

61


EuropeanUniversityAssociationasblAvenue de l’Yser 241040 Brussels, BelgiumTel: +32-2 230 55 44Fax: +32-2 230 57 51www.eua.be


The European University Association (EUA) is the representative organisation

of universities and national rectors’ conferences in 47 European countries.

EUA plays a crucial role in the Bologna Process and in influencing EU policies

on higher education, research and innovation. Thanks to its interaction with a

range of other European and international organisations EUA ensures that the

independent voice of European universities is heard wherever decisions are

being taken that will impact on their activities.

The Association provides a unique expertise in higher education and research

as well as a forum for exchange of ideas and good practice among universities.

The results of EUA’s work are made available to members and stakeholders

through conferences, seminars, website and publications.

examining quality culture part ii

Documents