university lecturers' experiences of ... - ujcontent.uj.ac.za
TRANSCRIPT
UNIVERSITY LECTURERS' EXPERIENCES OF THE SELF-EVALUATION
OF THEIR ACADEMIC PROGRAMMES
by
NICOLENE MURDOCH
Research Essay
Submitted in partial fulfillment of the requirements for the degree
MAGISTER PHILOSOPHIAE
in
ADULT AND COMMUNITY EDUCATION
in the
FACULTY OF EDUCATION AND NURSING
at the
RAND AFRIKAANS UNIVERSITY
SUPERVISOR: Prof SJ Gravett
January 2002
ACKNOWLEDGEMENTS
- I could not have done this without the following people:
my husband Wilhelm - you complete me;
my family and friends - for always inspiring and supporting me;
Prof Saartjie Gravett - for being such an inspiration and the embodiment
of an adult educator;
all my colleagues at the Center for Higher Education Studies - for always
supporting me and showing an interest in my research;
Dr Ton Vroeijenstijn for sharing his profound knowledge about the
subject;
all the RAU participants in this project who created a friendly and co-
operative working environment;
for the talent and persistence I received from the Creator.
ABSTRACT
The Founding Document of the Higher Education Quality Committee (HEQC) (January,
2001) stresses the importance of quality and accountability in South African higher
education institutions. These institutions will no longer operate in isolation, but are
accountable to governmental bodies. Quality assurance is considered to be an
institution's responsibility, but is not merely an internal activity. The Founding
Document clearly indicates that "once the HEQC is satisfied that demonstrable quality
assurance capacity has been established across a spectrum of higher education
providers, it will use a 'light touch' approach to quality assurance, based on an
increasing measure of reliance on the self-evaluation reports of providers" (HEQC
Founding document, 2001:15). Based on the above, it is clear that self-evaluation
reports of institutions will be of major importance in their quest for accountability.
The HEQC will thus to a large extent, depend on self-evaluation by institutions for the
orderly execution of their responsibilities.
In light of these recommendations, the RAU (Rand Afrikaans University) conducted a
self-evaluation pilot project involving three academic departments. The self-
evaluation that was conducted was not merely the collection of data, fact finding or
check listing, but a frequent, critical self-analysis of strengths and weaknesses of
academic programmes that should "lead to measures which can be taken in order to
improve quality" (Vroeijenstijn, 1995:51). The main purposes of such a self-evaluation
are improvement and accountability. Unfortunately, however, the self-evaluation
process is not without controversy and disagreement. Academics respond to this
process in different ways, and many feel that it infringes on their academic freedom.
The general aim of this study was to explore the experiences of academic staff at RAU
who were participating in the quality assurance self-evaluation process. This
qualitative study employed in-depth interviews and participant observation as data-
collection methods. The data was analyzed by using the constant comparative method
of data analysis.
This study found that there are a wide variety of responses among academics
regarding the self-evaluation process. Most of the lecturers realize the value of and
need for such an exercise, but there are a number of internal factors (e.g. lack of
visible management support, availability of reliable data) and external influences (e.g.
requirements of national bodies) that can hinder the process. The academics also
considered it to be a time-consuming exercise, but expressed their appreciation for
the provision of a structured manual that served as a guideline. They did,
nevertheless, stress the need to adapt and organize the exercise, provided in the
manual, to their individual needs.
Since the national education agenda demands that self-evaluation form an integral
part of the transformation and development of higher education, there exists an
urgent need to assess and explore the validity and reliability of different self-
evaluation methods. This study hopes to contribute to the present body of research by
focusing on the responses to such an exercise at the RAU.
TABLE OF CONTENTS Page
TITLE PAGE
ACKNOWLEDGEMENTS
ABSTRACT
INTRODUCTION 1
STATEMENT OF THE PROBLEM AND RESEARCH QUESTION 2
AIM OF THE STUDY 3
ASSUMPTIONS AND PRESUPPOSITIONS 3
REVIEW OF RELEVANT LITERATURE 4
5.1 Introduction 4 5.2 Quality in Higher Education 5 5.3 Quality, improvement and accountability 8 5.4 Self-evaluation for improvement or accountability 9 5.5 Self-evaluation: An international perspective 11 5.6 Self-evaluation: A national perspective 12 5.7 Institutional responsibility 13 5.8 Experiences of self-evaluation 14 5.9 Conclusion 16
RESEARCH DESIGN AND METHODOLOGY 16
6.1 Research paradigm 16 6.2 Setting of the inquiry 17 6.3 Sampling procedures 19 6.4 Data collection methods 20
6.4.1 In-depth interviews 20 6.4.2 Participant observation 22
6.5 Data analysis and presentation 23 6.6 Validity and reliability 27 6.7 Ethical considerations 30
DISCUSSION OF FINDINGS 30
7.1 Visible management support and institutional culture 30 7.2 Influence of contextual factors 31
7.2.1 Internal factors 32
7.2.2 External factors 33 7.3 Clarity of outcomes, goals and importance
of self-evaluation 34 7.4 Provision of an evaluation design 38
RECOMMENDATIONS 39
SUGGESTIONS FOR FUTURE RESEARCH 44
CONCLUSION 45
LIST OF REFERENCES 47
LIST OF TABLES Table 6.1 Categories, sub-categories and outcome statements 26 Table 6.2 Strategies used to ensure internal validity 28
LIST OF DIAGRAMS Diagram 10.1 Ripple effect of self-evaluation
46
ADDENDUM 1
Transcript of an interview ADDENDUM 2
Example of field notes
UNIVERSITY LECTURERS' EXPERIENCES OF THE SELF-EVALUATION
OF THEIR ACADEMIC PROGRAMMES
1. INTRODUCTION
Quality in higher education is becoming increasingly important for all institutions, not
only in South Africa, but all over the world (Lewis a Smith, 1994:2). It is not only an
institution's responsibility to maintain and improve internal quality and exhibit public
accountability, but it is evident that it is also on the agenda of the national bodies
such as the Council on Higher Education (CHE) and the South African Qualifications
Authority (SAQA). There are, however, several factors influencing (and sometimes
hindering) the quest for quality in higher and further education, such as striving for
international recognition or maintaining African relevance, the rising concern about
public expenditure, massification of universities, clarifying of institutions' purposes,
the rapid growth of technology and academic mobility, to name but a few (Lategan as
cited in Strydom, Lategan it Muller, 1997:75-76). Harvey (1995:12) stresses this
demand for increased accountability by saying "the quality agenda will continue to
grow as the nature and purpose of higher education shift to reflect the rapidly
changing world economy". This challenge, to replicate the economy, involves
rendering some form of account that the educational activities are being carried out
effectively and efficiently. It also implies that universities are moving from the "ivory
tower" image, to a much closer relationship with government and all the other
stakeholders involved (Lategan as cited in Strydom et al. 1997:78). This relationship of
accountability will attract increasing attention as competition between institutions
becomes more severe.
One of the distinct features that will elevate one institution above the rest, is by
marketing and delivering excellent programmes'. To enhance the quality of
programmes offered by providers, self-evaluation by staff members can prove to be an
extremely valuable tool. This evaluation involves an academic department critically
scrutinizing its own programmes and outputs, assessing the satisfaction of its
In this context a "programme" refers to a number of learning units (modules) leading to one or more qualifications.
1
stakeholders and taking corrective actions. The Higher Education Quality Committee
(HEQC) was instituted by the CHE to take responsibility for, among other duties,
external accreditation and evaluation of public providers and learning programmes,
monitoring of certification procedures, auditing and institutional reviews, quality
promotion and providing public information of governmental bodies. The Founding
Document of the HEQC (January, 2001) stresses the importance of quality and
accountability in South African higher education institutions. "Quality is identified as
one of the principles that should guide the transformation of higher education,
together with equity and redress, democratization, development, effectiveness and
efficiency, academic freedom, institutional autonomy and public accountability"
(HEQC Founding Document, 2001:1). It is clearly indicated that "once the HEQC is
satisfied that demonstrable quality assurance capacity has been established across a
spectrum of higher education providers, it will use a 'light touch' approach to quality
assurance, based on an increasing measure of reliance on the self-evaluation reports
of providers" (HEQC Founding document, 2001:15). Based on the above, it is clear
that the HEQC, as national quality assurance body, regards self-evaluation of academic
programmes as being of, major importance. They will utilize it when executing their
function of evaluating learning programmes of providers. This study focus on the
experiences of academic staff at the Rand Afrikaans University regarding the self-
evaluation of their own academic programmes.
The first section of this research essay focuses on the statement of the problem and
the aim of the study. This is followed by a review of relevant literature that informed
the research. Subsequently, the research design and methodology will be discussed.
Thereafter, the findings will be presented, possible recommendations will be made,
limitations of the study will be stated and possible topics for future research will be
suggested.
2. STATEMENT OF THE PROBLEM AND RESEARCH QUESTION
The Rand Afrikaans University (RAU) is currently busy with a self-evaluation pilot
project, involving three academic departments. The University has contracted an
international expert of the stature of Dr Ton Vroeijenstijn to help establish this
process of self-evaluation at the university. Dr Vroeijenstijn is the senior policy advisor
2
of the Association of Universities in the Netherlands (VSNU), with external quality
assurance as primary responsibility. According to Lategan (as cited in Strydom et al.
1997:593), self-evaluation is dealing with "the major issues of an institution. It
reflects the 'story' of the university and the 'hermeneutics' to understand the story".
It is not merely the collection of data, fact finding or check listing, but a frequent,
critical self-analysis of strengths and weaknesses that should "lead to measures which
can be taken in order to improve quality" (Vroeijenstijn, 1995:51). Self-evaluation can
be conducted for the purpose of internal improvement or as providing evidence for
external accountability. The self-evaluation process itself is however, not without
controversy and disagreement. Academics experience this process in different ways,
and often feel it infringes on their academic freedom. Due to these negative
perceptions, it is necessary to present the self-evaluation of programmes in a positive
light. Academic staff should realize that the exercise will not only result in personal
self-improvement, but also elevate the standards of the institution.
In light of the above, the research question can be formulated as follows: How did the
academic staff at the Rand Afrikaans University experience the quality assurance self-
evaluation process?
AIM OF THE STUDY
The general aim of this study was to explore the experiences of academic staff at RAU
who participated in the quality assurance self-evaluation process.
ASSUMPTIONS AND PRESUPPOSITIONS
I have been a permanent staff member at RAU for the past two years. I am primarily
involved in the quality related initiatives of the University. This portfolio is located in
the Center for Higher Education Studies. I have been specifically involved in the
administration and initiation of the self-evaluation exercise, where I developed the
perception that many academics do not respond positively to evaluative processes. I
have presupposed that academics experience this controversial exercise in different
ways, often feeling it imposes on their academic freedom. Attending numerous quality
assurance workshops and conferences where the issue of self-evaluation in higher
3
education was presented in a negative light, strengthened the latter assumption.
Collaborating and conversing with colleagues from other institutions who are also
involved in self-evaluation, created the impression that lecturers experience it as just
another regulatory exercise imposed by government. Consequently, I assumed that
they are not aware of the value and importance of self-evaluation for the purpose of
personal and departmental performance. Therefore, I presupposed they also do not
realize the importance thereof for institutional improvement and accountability.
5. REVIEW OF RELEVANT LITERATURE
5.1 Introduction
Changes in the environment and context of higher education are evident and rapid.
These changes are a worldwide phenomenon and arise due to the increasing
competitiveness among institutions, rising stakeholder expectations, and the challenge
institutions face having to "accomplish more with less" (Lewis Et Smith, 1994:x).
Ultimately, however, the important question to be addressed regarding the changing
landscape of higher education, is how members of the academy are responding to
these and other related developments. Lewis and Smith (1994:x) are of the opinion
that academics respond in one of two ways. On the one hand, they can react in a pro-
active and quality-driven manner, or, on the other hand, be defensive and
continuously strive to retain the past at the expense of the future. The latter would
result in failure and decease of constituencies holding such conventional views. Pro-
active institutions would recognize the need for continuous improvement and
assurance of the quality of products and services delivered.
Quality assurance or improvement is however a new and somewhat enigmatic concept
in higher education. Lewis and Smith (1994:2) support this view by saying "the
perception of quality in higher education is increasingly becoming a problem." Higher
education has a longstanding history and tradition of allowing individuals to function in
their own right and encouraging academic freedom and individuality. The challenge is
to change mindsets and misconceptions about the necessity of quality related
activities in higher education, even more importantly, to view the inevitable change as
an opportunity and not a threat.
4
Firstly, I will discuss the origins and notions of quality as derived from industry and its
appearance in higher education. Following on this, the focus of the literature review
will move to the issues of accountability and improvement as purposes of quality
assurance. I will then briefly describe the international and national history and
current state of affairs regarding self-evaluation and elaborate on the institutional
responsibility for self-evaluation. Finally, I will discuss some staff responses regarding
self-evaluation, as derived from the literature.
5.2 Quality in higher education
To investigate the "grand entrance" of quality into the field of higher education, we
need to look at its origins in industry, as it is from industrial practices that we derive
the basic concepts and methodology regarding quality (Sallis, 1996:8). Prior to and
early in the 1900's, quality was viewed as an integral element of craftsmanship and it
was controlled by supervisors. They were involved in a process of quality control once
the product was completed. This quality assessment moved towards a more
inspection-based and controlled process, measured against pre-determined standards
in the mid 1900's. Formal quality departments were only established in industry after
1960, and the concept of "total quality management" was introduced after 1980. This
concept proposes a practical and strategic approach to quality by focusing on the
needs of the customers (Sallis, 1996:28). Presently, it has expanded even further, to
include a culture of continuous improvement and organisation-wide quality
management and responsibility.
Regarding the educational context, only a few references to quality management are
found in the literature of the late 1980's. Quality management was initiated in both
the United States and Britain from 1990 onwards. Nationally, some formal structures
regarding quality assurance were only instituted from 1996 onwards (some
international and national perspectives will be elaborated on later on in this section).
Sallis (1996:10) is of the opinion, however, that there has been a "traditional
reluctance" in education to embrace, accept and adapt quality-related methodologies
associated with industry. Even though it is not ideal to relate educational processes
5
directly to manufacturing products, academics seem to be more susceptible to the
concept of quality assurance from lessons the service industry learned tong ago.
The problem of relating education to industry is even more complex because of the
difficulty of defining and measuring the quality of the service offered and the product
produced within an educational context. Not only are there large numbers of internal
customers involved (e.g. learners, lecturers etc.), but the needs of a variety of
stakeholders (e.g. parents, labour market etc.) have to be considered in the course of
the educational process. Despite the complexity of enhancing and promoting quality
services and products in education, there is a real need to address the issue because
"increasingly in education, quality makes the difference between success and failure"
(SalLis, 1996:1).
To make this difference and be perceived as striving to offer and maintain the highest
educational standards, we need to define quality in the context of higher education.
Quality experts around the world agree that quality is, like beauty, in the eye of the
beholder. It means something different for each person. Therefore, the former Quality
Promotion Unit (QPU) (instituted by the South African Universities' Vice-Chancellors'
Association) promoted the idea of a notion of quality, rather than an abstract,
simplistic, one-dimensional definition of quality (Brink, 1997:5). Within the context of
RAU, the Quality Care Committee (QCC) approved the following notions of quality:
Fitness of purpose refers to whether the mission, vision, values, objectives
and derived statements of the University comply with the dictates of national
policy, and whether they measure up to the expectations of recognised internal
and external stakeholders.
Fitness for purpose measure whether relevant attributes of the University
against the explicit and implied dictates of recognised University values, as
embodied in the mission and related statements.
Quality in the transformative sense focuses on the extent to which the primary
activities of teaching and learning, as well as research, develop the
competence of individual learners for personal enrichment, as well as the
requirements of social development and economic and employment growth
6
Value for money refers to whether the results of the various activities in which
the University and its subsystems are engaged warrant the resources allocated
to their conduct.
Quality means excellence, which implies striving towards exceptionally high
standards.
RAU decided to adopt the above-mentioned notions of quality, to stress the
importance of a common understanding when referring to certain phrases. It is
common practice internationally to use the generic term "quality assurance" to
describe all quality related activities. Vide, the name chosen by the international
organization for quality in higher education, called the International Network of
Quality Assurance Agencies in Higher Education (INQAAHE). Locally, the Higher
Education Quality Committee (HEQC) also uses the term in their Founding Document
and other documentation to denote all quality related activities. It also naturally
relates to industry, where quality assurance refers to ensuring that the production
system is under statistical surveillance for conformance to specifications, at certain
critical control points. These processes ensure that the final product also conforms to
original specifications and that a consistent conformance to those specifications can
be achieved. It is this constant conformance to specifications or, more specifically,
the constant deliverance of high quality service that should appeal to lecturers from
higher education institutions and should justify the implementation of quality
assurance systems. At RAU, however, it was decided to adopt the term "quality care"
for quality related activities, a term translated from the Dutch "kwaliteitzorg"
(Vroeijenstijn, 1990).
Quality care, in this context, includes four components, namely quality assurance,
quality promotion, programme accreditation and customer care. This "softer" term
was deliberately chosen to convey the impression that the University's quality
structure is not established to police the deliverance of quality service on campus, but
is there in a supportive role. The final responsibility for quality remains in the hands of
the responsible line-functionaries.
Quality assurance, first of the four components, includes all activities that are
intended to ascertain whether legitimate expectations are being met, whereas quality
7
promotion addresses issues that require improvement. Programme accreditation refers
to the verification that minimum quality criterion for accreditation by relevant
authorities, professional councils, regional, national and/or international bodies are
being met and customer care directly focuses on effective and efficient service to all
its customers by the University and its constituent parts.
To summarise, the aforementioned definitions suggest that the higher education
system aims to capture from industry what applies to their specific context in
education and training. It is thus evident that educational institutions should
increasingly become involved in quality assurance in order to contend with rising
competitiveness and stakeholder expectations. Furthermore, it is a professional
responsibility to strive for academic excellence.
Barlosky and Lawton (1995:51) propose four quality imperatives as driving and
motivating forces that challenge institutions to take up the quest for quality. Firstly,
there is the moral imperative, which claims that customers deserve the best possible
education. Secondly, there is the professional imperative of striving for high standards
and commitment. Thirdly, the competitive imperative focuses on the needs of
customers and lastly, the accountability imperative, which implies they must meet
the political demands for education to be more accountable and publicly demonstrate
the high standards of their products (e.g. curriculum, learning, graduates) and
services" (Sallis, 1996:5). For education, as for industry, improvement and
accountability is no longer an option; it is a necessity.
5.3 Quality, improvement and accountability
Improvement and/or accountability are the primary purposes of quality assurance. It
may seem, however, that external bodies, such as the CHE, would determine and
enforce these purposes. The willingness to improve and to be accountable is, however,
ultimately the responsibility of the institution itself. Thus, the institution is the
determinant of its own quality. This sense of responsibility and the task of alleviating
possible public misconceptions by not offering quality education is the institution's
prerogative. On the one hand, improvement is a matter of institutional integrity and is
only possible with constructive cooperation. On the other hand, accountability
8
"involves rendering some form of account that an activity is being carried out
effectively and efficiently" (Loder, 1990:2).
These two purposes are often seen as incompatible, but they are not mutually
exclusive. The challenge, according to Brennan, Frazer and Williams (1995:5), is to
achieve clarity and conformity as to the equilibrium that is sought between
accountability and improvement. Trow (1996:2) has written extensively on
accountability. He defines it as "the obligation to report to others, to explain, to
justify, to answer questions about how resources have been used and to what effect."
The extent to which accountability exists within an institution is determined by the
organisational structure, its culture and procedures, and where decision-making
powers lie. Wellman (2001:48), sees accountability systems in terms of "state-level
indicators of institutional performance, designed to reach public audiences, using
quantitative and qualitative measures that allow comparisons among institutions." It is
also called benchmarks or performance indicators.
An institution can however benchmark itself against internally set standards for the
purpose of improvement. Therefore, quality assurance can merely be aimed at
internal improvement. It can be achieved by identifying weaknesses and strengths in
the deliverance of academic programmes. Improvement may then involve some form
of change. This change includes building on the positive aspects and diminishing or
correcting the weaknesses that have been identified. These changes are not always
linked to increased financial costs, but an improvement-based approach can actually
save costs by identifying duplicated or wasted efforts, and proposing ways to eliminate
or reduce them (Open University, 1999). Sometimes, it does not involve any monetary
costs, but only means changing thinking processes or attitudes about correcting
previously unsuccessful activities. The increasing external pressures for accountability
in Higher Education necessitate urgent attention to the internal quality improvement
activities of all institutions. The quest for quality is equal to a quest for continuous
improvement.
5.4 Self-evaluation for improvement or accountability
9
As mentioned previously, one of the aspects of quality care being addressed at RAU, is
quality assurance. Self-evaluation is an activity primarily aimed at assuring the quality
of academic programmes. It is therefore an action intended to monitor and ensure the
levels of quality maintenance and identify aspects that can be improved upon. The
process of self-evaluation is not new, it has been an ongoing process at academic
institutions for a long time, maybe using different terminology. The notion of self-
evaluation is consistent with a range of educational, psychological and management
philosophies, such as 'the reflective practitioner' (e.g. Schon, 1987), 'metacognition'
(e.g. Biggs, 1991) and 'total quality management in education' (e.g. Sagor it Barnett,
1994 and Sagor, 1996). All of these scholars emphasize the capacity of individuals and
organisations to examine and evaluate their own behavior and actions, in order to
improve performance (Hall, Woodhouse a Jermyn in Strydom, et al., 1997).
In the literature, the terms "self-evaluation" and "self-assessment" are used
interchangeably. In higher education, the term "self-evaluation" is more commonly
used. It can be explained as follows. The use of the term "evaluation" in higher
education means to discover by investigation how well a specific educational approach
or activity is functioning. We can also distinguish between internal and external
evaluation. "Assessment" is currently used as measuring student learning outcomes. In
terms of quality assurance, "assessment" is used to include self-evaluation as well as
peer review (Open University, 1999). According to Brennan, Frazer and Williams
(1995:5) "self-evaluation is about whether educational objectives are being achieved
and whether current practice can be improved upon."
In an academic context, the question arises as to whether an institution should embark
on self-evaluation merely for improvement purposes, or should also be driven by the
requirements of accountability. Tension is caused between these two purposes,
because responses are influenced and determined by the purpose of the exercise. A
research project undertaken by the University of the Free State in 2000 concluded
that institutions viewed the two purposes of self-evaluation as of equal importance
(Fourie, 2000:7). The researchers concluded by saying that internal quality assurance
(aimed at improvement) "cannot retain sufficient momentum" without some kind of
external quality assurance (aimed at accountability). The New Jersey Commission on
Higher Education (2000) supports this view by saying: "Accountability and improved
10
performance are closely linked. Today's knowledge-based, global economy and society
hold extremely high expectations for colleges and universities and their graduates.
The challenge at hand demands open communication, the broad involvement of
stakeholders, pertinent information about performance, and a commitment to
improvement."
In light of the above, it is clear that the tension between the two purposes cannot be
ignored. According to Woodhouse (1995), this tension should be manageable within the
institution, since accountability and improvement have been simultaneously present in
the academic environment from the earliest times. The fact remains, however, that
the confidentiality of an internal self-evaluation report, and external public reporting,
for the purpose of accountability, will remain strenuous. Brink (as cited in Strydom, et
al., 1997), foresee that accountability will be formally built into the institutional
quality assurance systems of institutions in future. Nevertheless, it is still uncertain
how this accountability will manifest itself.
5.5 Self-evaluation: An international perspective
The process of self-evaluation has been part of the United States accreditation system
for nearly a century. Critical self-analysis has also been a fundamental element of the
United Kingdom's Council for National Academic Awards validation and accreditation
procedures for over 25 years (Open University, 1999). National quality assurance
systems operate in European countries such as Denmark, England, Finland, France,
Netherlands, Scotland, Sweden and Wales, and outside Europe. Other countries such
as Australia, Hong Kong and Mexico all have a process of self-assessment as a central
part of their quality activities and procedures.
In the United Kingdom, the Higher Education Quality Council (HEQC) was established in
1992 by universities and colleges in order to contribute to the improvement and
maintenance of quality. The emphasis of quality assurance in the United Kingdom has
been on improvement rather than accountability. Internal validation and review were
well established prior to any external monitoring, which means that institutions had
certain procedures in place prior to external audits (Geall, Harvey >t Moon in Strydom
11
et al., 1997:186). Currently, self-evaluation is encouraged and also the enhancement
of quality by sharing good practices.
Together with the United Kingdom and France, the Netherlands was in the "first
wave" of European countries to introduce self-evaluation in higher education
institutions (Westerheijden as cited in Strydom, et al., 1997:259). Much has changed in
institutions in the Netherlands, and the evaluation of quality played a major role in
those changes. Transparency of higher education institutions increased and a
significant change in Dutch academic culture has become evident. It is customary that
not all institutions are equal, and the bureaucratic "axiom of homogeneity" can be
loosened. According to De Haan and Frederiks (as cited in Strydom, et al., 1997:518),
the primary goal of the Dutch evaluation system, is improvement.
The Australian government states clearly that quality assurance and improvement
should be integrated into the strategic planning activities of universities. This is
ensured, among other methods, by institutional self-evaluation, followed by annual
consultations with the Educational Profiles (Candy ft Maconachie in Strydom et al.,
1997:334). It is evident from the literature that self-evaluation is internationally well-
known and widely utilized.
5.6 Self-evaluation: A national perspective
In South Africa, the change in the political dispensation in 1994 has resulted in
increased access to higher education, demands for the redress of past inequalities,
and calls for curriculum change to increase relevance in the new order. The major
development that necessitated that higher education respond to quality assurance
demands, was the establishment of the legislative environment, that lead to the
development of the National Qualifications Framework (NQF), as administered by the
South African Qualifications Authority Act (SAQA). According to Muller (In Strydom, et
al., 1997:38), the NQF represents a "state-initiated, state-controlled and state-
coordinated system of quality assurance that poses challenges to the traditional
academic freedom and institutional autonomy of higher education institutions."
12
The problem that has faced higher education in South Africa for the past few years, is
the absence of an external quality assurance agency for South African universities. The
Quality Promotion Unit (QPU) of SAUVCA (South African Universities' Vice Chancellors'
Association) functioned for a short period (1996 - February, 1999) (Fourie, 2000:3), but
was terminated. This situation was addressed in the Draft White Paper (Department of
Education, 1997), which recommended the formation of the Council for Higher
Education (CHE). The Higher Education Quality Committee (HEQC) was instituted by
the CHE at the beginning of 2000. In their Founding Document (January, 2001) the
importance of quality and accountability in South African institutions was stressed.
"Quality is identified as one of the principles that should guide the transformation of
higher education, together with equity and redress, democratization, development,
effectiveness, and efficiency, academic freedom, institutional autonomy and public
accountability" (HEQC Founding document, 2001:1).
It is evident from the literature, that the practices of self-evaluation in South African
universities, largely comprises of a combination of informal, traditional practices,
selectively applied, as determined by the institution. National developments and the
institution of the HEQC as national quality assurance body, the introduction of
external accountability and programme and provider accreditation necessitate regular,
structured self-evaluation at universities.
5.7 Institutional responsibility
It is evident that enhancing quality is an institutional responsibility, but not however,
merely an internal activity. The locus for responsibility in institutions for quality
assurance is continually under debate. The presence of a variety of professional
councils leaves the impression that they are taking responsibility, but the ownership of
quality must be shifted from the external body to the institution itself. Fourie (2000:3)
supports this view by postulating: "regardless of the existence of activities of any
external quality assurance agency, the major responsibility for systematic, structured
and continuous attention to quality, rests with the institutions themselves." This view
is also supported by the Department of Education (1997:22), as they state in the White
Paper on Higher Education that "the primary responsibility for quality assurance rests
with higher education institutions."
13
However, external systems do influence internal quality assurance activities. The
internal quality management system is determined by external requirements. The
National Commission on Higher Education (NCHE), recommended in its report that,
"Quality is not only an internal institutional concern, but also an essential ingredient
of a new relationship between government and higher education... A comprehensive,
development-oriented quality assurance system provides an essential mechanism for
tackling differences in quality across institutional programmes" (NCHE, 1996:11-12). It
is therefore essential that institutions take cognizance of governmental developments
and requirements when planning and executing internal quality systems.
5.8 Experiences of self-evaluation
Unfortunately, the self-evaluation process is not without controversy and
disagreement (Kimball, 2000). Academics' responses differ widely. On the one hand
they often feel it imposes on their academic freedom and on the other, they realize
the personal and institutional benefits thereof. Current literature reflects some of the
difficulties encountered when introducing self-evaluation processes in higher
education.
According to Frazer (1992), it is never easy to be self-critical and reflective. Self-
evaluation requires a constant, structured self-review. However, it is still necessary to
acknowledge the inherent professionalism of academic staff. Academics are often
suspicious that evaluation will have personal implications (such as losing respect and
credibility amongst colleagues) and that it is measuring personal productivity and
judging them individually. There is often a lack of understanding that it is aimed at
evaluating the system, and not assessing individuals. The lack of, firstly, awareness of
a systems approach to evaluation and, secondly, a quality culture in an institution, can
create these misconceptions. Poor communication of quality assurance, specifically
regarding the self-evaluation process, may lead to a loss of legitimacy for the system
(Jacobs in Strydom et al., 1997:165). Relevant information and good practices should
be disseminated through an effective and open system of communication.
Vroeijenstijn (1995:67) supports this view by saying that "a good self-assessment
(should be) as open and honest as possible."
14
Furthermore, self-evaluation is often experienced as an extra burden and
administrative responsibility, which academic staff feet should not reside with them.
On the other hand, centralized responsibility for regulatory exercises within the
institution creates a lack of commitment and involvement from staff. Staff often
either shift the responsibility, or are unable to recognize the need for self-evaluation.
Jacobs (as cited in Strydom, et al., 1997:155), enhances this view, by saying that
"staff in higher education may see quality as 'that person's job' and distance
themselves from it." He is of the opinion that staff should have ownership of the self-
evaluation process and participate fully in this collegial process, to avoid the above
misconception. These misconceptions are exacerbated when academics perceive the
process as being imposed upon them. Negative responses are often due to a lack of
transparency regarding the goals and aims of the process.
Bamett (1992:131) also argues that frustration is caused by a lack of structure or
standardized procedures regarding self-evaluation. Lecturers feet that their efforts
and the time invested in preparing the self-evaluation report are in vain, because it is
often not followed-up and there are no well-designed, continuous feedback processes
are in place. Many also claim that measures that are taken as a result of the self-
evaluation process are not clear, and that the entire experience is merely a paper
exercise. Furthermore, the absence of a quality management system or a specific
body, unit or committee leading the process, produces a perception of lack of
leadership and initiative. As mentioned earlier, staff often only experiences the
benefit of the self-evaluation in retrospect, as the following comment suggests, "It is
an anxiety-producing process that is not always easy, but it was good for us" (Kimball,
2000).
A number of suggestions are made in the literature to counteract these negative
perceptions and to better manage self-evaluation exercises. Brennan, Frazer and
Williams (1995:Preface) postulate that if self-evaluation is done only for public
relation considerations, it then develop into self-deception. This links with the idea of
creating unrealistic expectations of improvements, which cannot be met. It should be
obvious to staff members that top management supports and is completely committed
to the process of self-evaluation, otherwise, staff will not take the evaluation
15
seriously. Not only should the university management be convinced of the need for
quality assurance and self-evaluation but financial and infra-structural support and
resources should also be made available. Staff often perceive it as a top-down
approach, which produces conflict regarding power relations (Open University, 1999).
5.9 Conclusion
To gain a holistic perspective of quality assurance related to the context of higher
education, and the role that self-evaluation plays in this regard, it has been necessary
to discuss the origins and notions of quality adopted in the context of higher
education, and in this instance, specifically RAU. Consequently, the current debate
regarding issues of accountability and improvement as purposes of quality assurance,
was discussed. To conceptualize the research it has been important to focus briefly on
the national and international scenarios regarding self-evaluation and to emphasize
the institutional responsibility for self-evaluation. Lastly, (strengthening the purpose
of this study) I focused on the limited amount of information in the literature and
often neglected aspect of self-evaluation, namely the experiences of academics.
In the next section I will discuss the research design and methodology used in this
inquiry.
6. RESEARCH DESIGN AND METHODOLOGY
6.1 Research paradigm
Each person lives life and experiences reality based on certain pre-constructed and
sometimes unconsciously present paradigms and frames of reference. Guba and
Lincoln (1989:80) defines a paradigm as "a basic set of beliefs, a set of assumptions
we are willing to make, which serve as touchstones in guiding our activities." This is
then the philosophical framework, which not only guides the lives and decision-making
of researchers as people living in the world, but more importantly in research,
determines the design, methods and methodology in which they choose to conduct
their research.
16
These choices of design, methods and methodology are also guided by the aim of the
study they are undertaking. In this case, the aim of the inquiry was to determine and
understand how academics experience the self-evaluation process of their academic
programmes. In determining experiences of the participants, the researcher, as well
as the participants, play an active role in constructing the reality regarding a specific
exercise or intervention (Guba a Lincoln, 1989:84). Thus, it can be postulated, that a
constructivist paradigm informed this research inquiry. There will necessarily be
differences in experience amongst both the participants and the researcher. My role as
researcher was then to be "the investigator" and therefore "the primary instrument
for gathering and analyzing data and (...) respond to the situation by maximizing
opportunities for collecting and producing meaningful information" (Merriam,
1998:20). I, as the researcher, then played an active role in constructing reality, by
walking the way with the participants, attempting to experience their world and
reality. Consequently, I used qualitative methods of inquiry, as these are preferred
methods within such a constructivist paradigm.
Since this was a qualitative study, it follows logically that I employed qualitative
methods of data collection. My main method of data collection involved in-depth
interviews with participants and, to a lesser degree, participant observation.
6.2 Setting of the inquiry
As mentioned previously, this study focused on lecturers' experiences of the self-
evaluation process of academic programmes. It was conducted at the Rand Afrikaans
University (RAU) located in the heart of Johannesburg. It is an urban university with
almost 20 000 students (about 6 000 are distance education students). The university
has six faculties, with a variety of 46 departments within the various faculties. Three
departments were involved in the self-evaluation process and they were located
within the Faculty of Education and Nursing, the Faculty of Science and the Faculty of
Economic and Business Management.
As explained in the previous section, self-evaluation is one of the primary activities of
the quality initiative at RAU. The quality portfolio is located within the Center for
Higher Education Studies at the University, implying that the Center is responsible for
17
administering the self-evaluation exercises. The quality portfolio consists of two
individuals (of which I am one) responsible for the quality related activities at RAU.
Due to this limited capacity, RAU has contracted an international expert of the stature
of Dr Ton Vroeijenstijn to help establish this exercise of self-evaluation at the
university. The reason for such an intervention will not only make a valuable
contribution to quality assurance at the institution by putting it on a sound foundation,
but it will also put RAU in a favourable position on national level. Dr Ton developed a
self-evaluation manual specifically for RAU, which was implemented for the first time
in the abovementioned three departments.
This manual was specifically developed for academic departments. However, self-
evaluation can (and should) be conducted in non-academic departments as well. For
the purpose of this study, I focused on the self-evaluation of academic programmes
within the three academic departments currently involved in the process. It is evident,
from national developments, that self-evaluation reports of institutions will become
increasingly important in their quest for improvement and accountability (as discussed
in the previous section). National bodies indicate that they would, to a large extent,
depend on self-evaluation by institutions for the orderly execution of their
responsibilities.
When evaluating academic departments, Vroeijenstijn (1995:52) is of the opinion that
self-evaluation should consist of the following subjects regarding the inputs, processes
and outputs of the department:
the place of the department in the organisational structure of the complete
system - the goals and aims of the department and its programmes should fit
the general mission and vision of the institution;
a description of the students, e.g. success and dropout rates;
description of the programmes in terms of content, organization, didactic
concept, curriculum design and assessment;
the educational process, e.g. teaching methods, utilisation of media etc.;
satisfaction of the students, alumni, labour market, society and staff.
18
Each of the above-mentioned subjects should be analysed by describing them in the
following manner:
a description of the current situation;
an analysis of the situation;
problem-solving - if the situation is not satisfactory, the way in which it can be
changed and improved should be described.
Through self-evaluation, departments should become aware of their strengths and
weaknesses, and start to restructure the weaknesses (Kells, 1995).
6.3 Sampling procedures
Miles and Huberman (1994:27) are of the opinion that qualitative researchers study
small samples of people, situated in their context, comprehensively. I used purposive
(or sometimes referred to as purposeful) sampling in my inquiry. This implies that "the
investigator wants to discover, understand, and gain insight and therefore must select
a sample from which the most can be learned" (Merriam, 1998:61). Thus, eight
academic staff members from the departments involved in self-evaluation were
selected purposefully, to ensure a variety within the sample (Maykut and Morehouse,
1994:45). Initially, I planned to interview nine participants, three from each
department, but in the fifth and sixth interview recurring patterns and themes
became evident, and after eight interviews, a point of redundancy (Guba ft Lincoln,
1989:64) was reached.
The selection of the sample was based on the "information-richness" (Patton, as cited
in Merriam, 1998:61) criteria, and furthermore, a combination of typical and
convenience sampling. Convenience sampling was utilized in terms of the participant's
time and availability for interviewing and typical sampling with regards to the
academics' active involvement in the process. The participants were not necessarily
on the appointed self-evaluation committee (to drive and manage the process within
the department), but in some instances, they were specifically chosen for not being on
the committee. This resulted in collecting opinions from both perspectives. Males and
females were represented almost equally, as well as, a variety of age groups and
academic positions.
19
Before the inquiry, all the relevant departments agreed to participate in the study.
Lecturers from the different departments were contacted individually, and
appointments were made. I experienced enthusiasm and willingness among the
lecturers to participate in the research project. The participants chose the setting
where the interviews were to be conducted which in all the cases were their own
office environments. Due to the familiar surroundings, most of the respondents felt
relaxed and confident to talk about their experiences. As the interviewer, I felt more
comfortable interviewing some of the respondents than others, because some of the
participants were much more willing to disclose very personal experiences, whereas
others were at first more hesitant. But after I explained my role as researcher and the
aim and purpose of the study, the respondents were willing to participate and
contribute in a co-operative manner.
6.4 Data collection methods
For the purpose of this study, as mentioned briefly, I have made use of in-depth
interviews and, to a lesser extent, participant observation. Using multiple sources of
information is valuable "because you want to use different methods ... to corroborate
each other so that you are using some form of methodological 'triangulation" (Mason
in Silverman, 2000:98). The use of in-depth interviews and participant observation, as
data collection methods, will now be discussed.
6.4.1 In-depth interviews
The primary method of data collection was personal interviews. An in-depth interview
takes the form of a social conversation between two people. De Vos (1998:300) is of
the opinion that an in-depth interview "enables an interviewer to obtain an 'insider
view'." Patton (1987:109-114) distinguishes between three approaches to qualitative
interviewing. One of the approaches is defined as the "informal conversation
interview" (also see Merriam, 1998:71). This approach relies entirely on the
spontaneous generation and formulation of questions within the natural flow of
interaction.
20
Eight academic staff members were interviewed individually. The reason for using
individual interviews as opposed to focus group interviews, was the objective of
eliciting truthful responses and eliminating the threat of a group situation. The length
of the interviews varied from individual to individual. The shortest interview took
approximately twenty minutes, and the longest, almost one hour. The interviews were
scheduled over a period of a month. The different departments were at different
stages of the self-evaluation process, which contributed to the "information-richness"
of the data (Patton, 1990:172). The interviews were conducted in the language of
preference of the participant.
During the preparation phase, before the interviews were actually conducted, I
studied the relevant literature and also the specific instrument for evaluation, as
developed by the consultant. Initially a total of nine interviews were tentatively
planned, but the actual number of eight was determined by the data and information
obtained from the various participants.
The interviews were planned in terms of three phases, namely the beginning, middle
and terminating phase. The beginning phase of the interview was informally structured
along the following broad lines. Firstly, I explained that confidentiality would be
maintained since the conversation took place only between two individuals. The
respondents were thanked for their willingness to contribute and it was also explained
that their input was regarded as valuable for further self-evaluation exercises in
future. As a researcher, I ensured the participant that no information would be
attached to them personally or to the department they represent in any way. I also
obtained all the respondent's permission to tape record the interview. A disadvantage
of recording interviews, which I also experienced, was that some of the respondents
are constantly aware of the recorder, and saw it as some form of "intrusion." I assured
the participants that the recording was merely for my own personal use, as I was not
able to make notes and remember all the important aspects discussed. After some
initial discomfort, in some instances, all the respondents expressed themselves freely
about their experiences.
The next step was to explain the goal and purpose of the study to the participants. It
was necessary to clarify my role as researcher, because I was also involved in
21
administering the self-evaluation process at the university at the time. It was stated
clearly that the purpose of the interview was not to determine the current status or
progress of self-evaluation within the department.
The middle phase of the interview was introduced by asking participants to tell me
about their experiences regarding the self-evaluation process. The rest of the
interview questions spontaneously developed as part of the interaction. As the
interviewer, I had a list of issues at hand to refer to, when specific direction was
needed. Probing was used to further clarify uncertainties and to get additional
information when the responses were not satisfactory. According to Nachmias
(1997:243), probes motivate respondents to elaborate or clarify responses or explain
reasons behind certain answers. These were derived from answers or comments to
previous questions. An example of a probe would be, "So, you hope that this exercise
will solve some of those problems?", following up on a comment the respondent made
about a solution to certain problems regarding academic programmes. Another would
be, "Are people actually involved?", responding to an unclear comment made about
the role of fellow academics.
The last phase was to terminate the interview, where the participants were thanked
for their availability and willingness to participate. I also mentioned the possibility of
future interviews and indicated that I would contact them if necessary.
An example of a transcription is attached as Addendum 1.
6.4.2 Participant observation
Participant observation was used as a secondary method of data collection. The
observations were done during the preliminary discussions and information sessions
prior to the self-evaluation exercise and during other supporting or information-
seeking informal conversations with the departments involved. Field notes were made
(an example of the field notes are attached as Addendum 2) when the consultant, Dr
Ton Vroeijenstijn, visited the six faculties in March 2001, and again when he visited
the participating departments in June of the same year. It was quite easy to take on
the role of observer, because I did not directly participate in the conversations
22
between the consultant and the members of the department. Consequently, this
created the ideal opportunity to observe in a non-threatening manner. Verbal
questions and concerns were raised, regarding possible problems in the future.
Another important aspect observed during these sessions, was the non-verbal
communication and messages conveyed by participants. This not only provided
background information regarding the structure and internal politics of the
department involved, but also about the attitude of the participants to the planned
exercise. Short notes were made during the sessions, which were expanded upon as
soon as possible after each session. According to Spradley (as cited in Silverman,
2000:141-142), this improves the reliability of the field notes.
The sessions were noted mainly (according to the checklist in Merriam, 1998:97-98) in
terms of:
the physical setting - naming the venue and time allocated for the session;
the participants - not individually labeled, but whether it was the complete
department, or only the appointed self-evaluation working group;
the conversation - I especially noted the inputs regarding the participants'
experiences and the information the consultant provided for clarification;
subtle factors - such as the attitudes or approaches where departments had to
introduce themselves to Dr Vroeijenstijn, non-verbal messages and evidence of
interest of members who were present.
The findings, from studying these field notes, strengthened and confirmed the results
that were derived from the interviews. I, as the researcher, had the advantage of
experiencing the process first-hand. However, according to Merriam (1998:95), this
contains an unreliability disadvantage, because it is based only on the perceptions of
the researcher. As with the interviews, this was a time consuming exercise, but
nevertheless made valuable contributions and advancements to the data collected.
6.5 Data analysis and presentation
De Vos (1998:336) states that analysis is "a reasoning strategy with the objective of
taking a complex whole and resolving it into its parts." It is the process of organizing
the data into more manageable categories. Bailey (1996:89) is of the opinion that data
23
analysis starts at the moment the research is thought of, and continues up to the final
report writing. Merriam (1998:178-179) on the other hand, explains the level of
analysis as "data (that) are compressed and linked together in a narrative that conveys
the meaning the research has derived from studying the phenomenon." The interviews
were all recorded and transcribed verbatim by myself for the purpose of analysis.
These transcripts were analyzed by means of the constant comparative method of data
analysis (as discussed by Maykut a Morehouse, 1994:126-144). This method consists of
a "continuous developing process" (De Vos, 1998:339) of comparing incidents and
searching for recurring themes and patterns (Merriam, 1998:159).
The data collection and data analysis were done simultaneously. I read through the
initial interview transcription and field notes made during the observed sessions. I
highlighted important pieces of information and made notes of main ideas in the
margin. I started again with the first interview and identified the smallest units of
meaning. This consists of the smallest sentence or paragraph that can stand on its
own, and still be meaningful and reveals information relevant to the study (Merriam,
1998:179-180). An example of a unit of meaning would be in one of the interviews, "I
definitely think it is necessary" and "It is time consuming." A unit of meaning does
not necessarily consist of only one sentence, or part of a sentence. Sometimes it is a
discussion of half a page regarding a specific idea. In the margin next to the first
example, I would write the need for self-evaluation, and next to the second example
negative experiences. Each unit of meaning was coded, to be able to know where it
was derived from. An example of a code would be BW1 /1. BW being an identity
assigned to a specific person, "1" being the first (of a possible two) interviews, and
lastly, reference to the page number of the transcription being analyzed. The
transcriptions were then photocopied and literally cut to pieces. Each unit of meaning
was cut out. These units were compared with each other then placed into preliminary
categories or classes as constructed through reading through the data. These tentative
categories were refined from the literature and during the data collection phase as
ideas were conceptualized and concretized. These categories related to the purpose
of the research. In further readings of the data (as interviews were conducted), the
notes were compared to what was derived from the first interview. Some of these
tentative categories were, for example:
➢ positive experiences;
24
negative experiences;
culture or awareness;
context;
management support and commitment;
the instrument etc.
Data that did not fit into one of the provisional categories, was categorized
separately, for later inclusion in a newly created or tentatively named category. The
names assigned to these preliminary categories were merely for the organization of
the units of meaning. Further development of the names of the categories was
constructed during the course of data analysis. The next phase was to compare and
integrate the categories. In some instances new categories were born, and others were
merged. Some of the categories were also renamed. A rule for inclusion was written
for each category to determine what would be included and what would be excluded
from that specific category. The units of meaning were scrutinized to determine
whether they "qualify" for inclusion within a specific category. These rules for
inclusion were refined and rewritten several times to finalize them. An example of a
rule for inclusion for the preliminary category "positive experiences" would be "Some
lecturers realize the value and need for such an exercise." As the interviews were
conducted, they were transcribed and categorized in the described manner until a
point of redundancy was reached.
Outcomes were prioritized according to the outcome propositions in light of their
importance in contributing to the purpose of the study and their prominence in the
data (Maykut Et Morehouse, 1994:158). These outcome statements produced the
foundation of the findings.
The following indicate the final categories, sub-categories and outcome statements:
25
Table 6.1: Categories, sub-categories and outcome statements
Categories and Outcome statement
subcategories
Visible management
support and institutional
culture
Management
commitment and
support
Institutional culture
There are uncertainties about the support and
commitment of management and therefore the success
of the self-evaluation exercise. This does not only
include moral support, but also practical, visible,
monetary and infrastructural support. A culture of
quality and a need for continuous improvement should
be established.
Influence of contextual
factors
External factors
Internal factors
Internal and external contextual factors influence the
self-evaluation. Lack of clarity of external requirements
regarding self-evaluation and other national initiatives
hinders the process of self-evaluation. Integrated,
reliable data availability is an essential aspect of
successful self-evaluation. Lack of co-operation of fellow
academics often hinders the quality of the process.
Clarity of outcomes, goals
and importance of self-
evaluation
The academic staff realizes the importance, necessity
and value of self-evaluation as an improvement
mechanism. Although the need for self-evaluation is
evident, academics still experience it as an additional
time consuming, frustrating and highly administrative
burden. The expectations, goals and consequences of the
self-evaluation were not clearly identified and
communicated.
Provision of an evaluation
design
A well-structured instrument needs to be provided as a
guideline, but academic freedom should still be
recognized and allowed in the adoption and tailoring of
the instrument for the specific context.
26
The efficacy of categories was determined by using the following guidelines, as
provided by Merriam (1998:183):
each of the categories reflected the purpose of the study;
the categories were exhaustive, in that they made provision for all data;
the categories were mutually exclusive - there was not a possibility of placing a
unit of data into more than one category;
the names assigned to categories were as sensitive and descriptive as possible;
they were all conceptually congruent, thereby indicating a similar level of
description in all categories.
The above-mentioned findings will be discussed in detail in the following section.
6.6Validity and reliability
It is necessary to provide evidence to prove that what the researcher has examined is
noteworthy and can be trusted. In the attempt to prove this, I investigated the data
collection and analysis for internal validity, external validity and reliability. By
following this form of investigation, confidence was established in the investigation
and results.
Merriam (1998:201, 207) distinguishes between internal and external validity. Internal
validity deals with the coherence between the results and the dynamic reality. She
(1998:204) proposes six strategies to enhance internal validity. The following table
indicates which of the six strategies were used in this study:
27
Table 6.2: Strategies used to ensure internal validity
Strategy Description of strategy Application to ensure
internal validity
Triangulation Multiple sources of data and
methods of data collection used
or the combination of one or
more research methods in the
one study (Myers, 2001).
For the purpose of this
study two methods of data
collection were used,
namely in-depth interviews
and participant
observation.
Member checks People that served as data
sources were consulted to
clarify uncertainties.
An open communication
channel was established
between the researcher
and participants, which
provided the opportunity
for regular member
checks.
Long-term observation Observation took place over a
period of time.
The study was conducted
over a period of one year.
There was opportunity for
development with regards
to academic staff's initial
and subsequent
experiences.
Peer examination This is referred to as "peer
debriefers" (Byrne, 2001), and
it is when the researcher
consults experts and colleagues
to ensure validity of
interpretations.
I had contact regularly
with the consultant, my
colleague and trusted
mentor, and colleagues at
other institutions who were
also in the process of self-
evaluation.
28
Strategy Description of strategy Application to ensure
internal validity
Collaborative mode of
research
Involving participants in all
stages of the research.
The participants were
observed in the beginning
stages, and interviewed at
the latest possible stage of
the project.
Researcher's biases Clarifying assumptions of the
researcher.
I had definite biases at the
onset of the project, but
after conducting my first
interview, this was
clarified.
External validity refers to the possibility of applying the findings to other situations.
Hoepfl (1997:13) describes this transferability to other situations as depending "on the
degree of similarity between the original situation and the situations to which it is
transferred." Researchers should therefore provide the reader with sufficient
information to prove the external validity or transferability to other situations.
Two strategies (as porposed by Merriam 1998:211) were used to aim for ultimate
external validity. Firstly, I provided, as far as possible, detailed information and
descriptions to help the reader to decide whether their situation is similar to the
described setting. Secondly, I described, where information was available, how other
institutions or departments experienced self-evaluation. I am of the opinion that it
would not only be transferable to similar (discipline-specific) departments, but also to
other departments involved in self-evaluation, due to the generic aspects of such an
exercise.
The problems regarding reliability in qualitative research are discussed by Hoepfl
(1997:13) and Merriam (1998:205). On the one hand it is problematic because human
behaviour is dynamic. On the other hand, qualitative researchers are often focused on
proving or providing evidence for validity, rather than concerning themselves with
reliability issues. In understanding reliability, Guba and Lincoln (as cited in Merriam,
29
1998:206), suggest the use of the terms "dependability" and "consistency." The
following techniques were used to ensure reliability by providing information
regarding:
the context, the researchers' position and the relationship with the selected
participants;
triangulation, as discussed previously and
providing a clear and defensible link for each step from the raw data to the
reported findings - in other words, a clear and logical "audit trail."
6.7 Ethical considerations
As explained earlier, as a researcher, I was fulfilling a two-fold role in the self-
evaluation project. On the one hand, I was part of the administration of the process,
and on the other hand, the researcher of the process. With regards to the above-
mentioned duality of roles, ethical considerations had to be taken into account
concerning the gathering of information in a non-threatening and non-"policing"
manner. I also had to be aware of using information obtained by means of interviews,
to the advantage or disadvantage of a specific department. Confidentiality had to be
ensured and roles had to be explained. The fact that no information would be
attached to individuals or specific departments had to be clarified. Codes were
assigned to each individual for use in the transcribed interviews, so that no person was
in any way linked to a specific interview.
7. DISCUSSION OF FINDINGS
7.1. Visible management support and institutional culture
Respondents do not consider the intentions, visible commitment and support from
management as being sufficiently clearly defined. Lewis and Smith (1994:285) are of
the opinion that any quality initiative can only succeed if there is visible commitment
from management. Management intentions are questioned in that they "should not
support the activity to look good in government circles" and that it should not be seen
as a control mechanism. They should show that they are serious about quality and
commit themselves, not only in terms of verbal agreement, but also in terms of
30
practical provision of essential resources. The interviewees acknowledge that it is not
always merely a matter of providing money. Sometimes, more significantly, a change
in attitude is required. But in the instances where money is needed, management
should provide financial support.
According to Dr Vroeijenstijn (2001a:4) the issue of the involvement of management is
a matter of international concern and importance. Self-evaluation depends, on the
one hand, on support from management, but, on the other hand, they cannot enforce
it, the responsibility and ownership should lie within the department. Kells (1995:6)
indicates that higher education institutions often do not display "vital management."
In addition, they are often hesitant to change, and characterized by reactive quality
management. This distinctiveness of academia creates "stumbling blocks to
formulating and completing a useful self-evaluation process and to initiating and
sustaining ongoing, internally motivated cycles of such activity" (Kells, 1995:6).
The respondents indicated that visible involvement of management is essential in
creating an institutionally wide culture for continuous improvement. This general
creation of awareness will result in the necessary sensitivity that ultimately creates
quality. Some faculties are more open to a culture of improvement, especially where
they have been exposed to accreditation of professional bodies on a regular basis.
Management and staff need to acquire a sound knowledge base of quality. National
and internal tendencies are instrumental in fostering this philosophy.
The respondents realize that the creation of a continuously improvement culture is a
long-term process. One of the participants mentioned, "I am not convinced in my
heart that they are completely committed to the process." Kells (1995:24) supports
this view by explaining that "building the culture will take time and, of course, much
progress can be achieved through a good experience with a self-evaluation process."
7.2 Influence of contextual factors
The respondents are of the opinion that contextual factors and influences must be
understood and analyzed to get meaningful participation with regards to self-
evaluation. They mentioned "we must be aware of national developments" and
31
"internally, changes and happenings should be communicated." They feel there should
not only be an internal consciousness of the organizational environment, but also an
awareness of the larger context regarding governmental and international trends
(Lewis Et Smith, 1994:2). Academics need to be informed about global and local
initiatives and tendencies, to indicate that what is expected of them is in line with
international practices. A distinction can be made between internal and external
factors influencing self-evaluation.
7.2.1 Internal factors
Each and every institution should take responsibility for quality in its own environment
and context. Knowledge of the organizational context includes the internal
relationship between departments and faculties, making comparisons and creating
realistic openness. Issues like internal restructuring and reformations affect the
success and feasibility of self-evaluation. There is also a definite need for South
African institutions to raise the levels of their internal quality, due to rising
competitive challenges.
In terms of the internal context, the participants experience the dedication and
commitment of co-academics as problematic and sometimes literally hindering the
process. This applies especially to those not serving on the departmental self-
evaluation committee. If staff members could not see the personal benefit of the
exercise, they did not do anything before some concrete contribution was expected of
them. The "passive observing and unwillingness to share information" and best
practices amongst departments are impeding the natural flow of the process.
Respondents felt that "people's hearts weren't always in it" and "they didn't go the
full nine yards." Another problem was a degree of dishonesty regarding the personal
information provided, which impacted on the accuracy of the report. Some individuals
felt they had to protect themselves and therefore did not disclose certain information.
Kells (1995:30) is of the opinion that "self-evaluation has often been seen to improve
communication patterns, trust, listening, and group functioning in facing and solving
problems." It should be used to the advantage of the department, and not have a
detrimental effect.
32
Insufficient provision and lack of reliable data was a foreseen problem within the
internal context of the institution was confirmed by the research results. This was due
to problems regarding the information management system operating at RAU. Kells
(1995:4) postulates that this is an international problem by claiming that, in general
the infrastructure needed to support planning, decision-making, self-evaluation and
regulatory activities, is weak. The respondents experienced frustration and a rupture
in the momentum because of the availability of fragmented, incomplete and outdated
data. They felt that there was nothing "reliable and overarching" available. Within
some faculties or departments their own initiatives with regards to data capturing
helped to some extent, but overall "it took a lot of time to collect all the data." This
frustration mounted because it is a hindering factor and the departments had no
control, and had to "constantly question the validity and reliability of the data". This
unfortunately shows deficiencies in the data management system of the institution,
because "information is not readily available." The tack of data also resulted in open
spaces, where answers were not available, which may create an incomplete picture.
7.2.2 External factors
Kells (1995:6) is of the opinion that institutions venturing into self-evaluation are not
only internally affected, but also by circumstances and aspects beyond its own
organizational borders. As mentioned in the problem statement, the Higher Education
Quality Committee (HEQC) will rely heavily on self-evaluation reports. It is definitely
problematic to continue with the process before "clear guidelines are provided and
established by the HEQC." However, the departments realize they cannot wait for
governmental interventions, they should get their own houses in order. The relevance
of self-evaluation against external changes and restructuring are always in question,
but inaction will not be beneficial to the institution.
Some departments are used to being accredited by and adhering to requirements of
external bodies and professional councils within the broader context, "we are used to
this, we've been doing it for years" and "for us, this is nothing new." According to
Webbstock and Ngara (in Strydom, et al. 1997:556) "external quality monitoring of
university programmes has been the preserve of professional bodies which accredit
career-oriented programmes." They have the power to determine minimum
33
requirements for students entering into certain professions. Dr Vroeijenstijn (2001a:4),
however, warns departments, involved in such procedures, to beware of only adhering
to the standards set by professional or other external bodies. It does not always
include an element of evaluation and analysis. They should be critical of what is
required and add on to that, in order to make a complete self-evaluation. Academic
standards are dynamic and not static, and must be adapted regularly to remain
internationally competitive and comparable.
7.3 Clarity of outcomes, goals and importance of self-evaluation
It was evident that the participants realize the importance, need and value of self-
evaluation. It was stressed that it is important and valuable for internal development.
Pekar (1995:38) supports this view by saying that "everyone can benefit from such an
exercise because no one can ignore themselves." The respondents indicated that in
most cases they were aware of the problem areas, and that the self-evaluation
provided a medium to communicate the realistic or real state of affairs. Some other
comments regarding the need and value of the exercised were expressed, for
example:
"I really think it is necessary."
> "I think it is a sound principle."
"The necessity of self-evaluation is evident."
"We need to continuously evaluate ourselves."
The respondents were of the opinion that you cannot rectify or improve on the current
situation if you do not know what your weaknesses are. One participant made the
remark: "It is after all self-evaluation for ourselves, it is not forced upon us by
management. They will not put a red cross next to your name." Academics generally
agreed that we should have started walking this road a long time ago, and that it is no
longer a question of if we should, but when and how we should embark on self-
evaluation. According to Dr Vroeijenstijn's experience, most departments that
exposed themselves to self-evaluation responded that although it is an extensive
exercise, they were always glad that they had engaged in it. The value of the exercise
always overshadows the negativity once the process is completed.
34
Due to my presuppositions and assumptions, as the researcher, I was gratified at the
positive responses to self-evaluation. One of the respondents stated "it made me
realize how important it is for us to think about what you want to achieve with your
programmes, and to what extent you are successful." They also indicated that it is
worthwhile to sit down and investigate and discuss the underlying coherence of course
contents. They were able to identify long overseen overlaps which provided for better
co-ordination of different modules in the future. The academic staff indicated that
they often fail to see the bigger picture and underlying relationships, which the self-
evaluation exercise highlighted.
They indicated that "it was not so bad as we thought it would be, as soon as you get
started you're fine." The importance of having a willing attitude to change and
improve was thought to be a vital part of what they considered to be a "learning
experience." With regards to handling the exposure to criticism, they indicated that
"you have to evaluate the merit of the criticism" and use it for improvement. They
realized that "it is not personal, it is a way to say how you are going to improve."
Lastly, one of the respondents indicated that "self-evaluation came at exactly the
right time when we realized we had to do something in terms of legislation to ensure
quality," which is also an indication of the realization of the importance of self-
evaluation.
The respondents felt that the goals and expectations of the exercise were not clear
and communicated thoroughly. Some insecurity was experienced because of the
unclear expectations:
> "It wasn't clear what was expected of us."
➢ "It was not completely clear. What are we aiming for? What are we trying to
get out of this?"
> "One of the problems was we did not know what the outcome will be."
Kells (1995:30) explains that it is difficult to align "staff members' perceptions of the
purposes, structure and activity of self-evaluation." It definitely emphasized the
importance of stressing the improvement goal of self-evaluation, and not just as part
of or preparation for governmental or external initiatives.
35
Even when the goals and needs for self-evaluation are clear, as with any type of
criticizing or regulatory exercise, there are some negative or problematic experiences.
First and foremost is the fact that self-evaluation is an extremely time-intensive
activity. Some academics felt that it is not the core business of the institution and
that it distracts them from time that should rather be spent on the core educational
activities.
Other negative experiences included that of frustration and resistance. Academics felt
it was "just another thing that landed on our desks." They were also not convinced
that the advantage of the exercise would be realized or filtrated to all members
involved. One respondent verbalized it as follows, "One danger I see is that this
exercise will not result in wider enrichment."
Again with regards to exposure to criticism, responses were: "Nobody likes to be
exposed," "It is never easy to be criticized." For some it was realty a "painful
process" to counteract the threat of criticism, Dr Vroeijenstijn advised that the
evaluation should by no means be individual in nature, but that a collective approach
should be followed.
The consultant, Dr Vroeijenstijn (2001a:6), explained from the outset that self-
evaluation would be a costly and time-consuming exercise. According to Kells
(1995:4), it is often expected of higher education professionals that are specialists in a
specific field, to perform in other areas where they often have "little or no training."
Frequently having to deal with an enormous workload, the self-evaluation was in some
instances seen as an additional administrative burden. Academics referred to the fact
that it was not this initiative specifically that they objected to. They had to deal with
a number of other national and internal initiatives and were having to cope with more
and more administrative tasks. They expressed this frustration and irritation in the
following manner:
> "If it wasn't for the heavy workload, we could have spent a lot more time on
it."
> "I thought, why on earth do we have to do this?"
> "Where are we going to find time to do this?"
> "I felt we are doing the job of business consultants."
36
Some academics, on the other hand, acknowledge that everyone must help to carry
the administrative load of the university. Others feel that their colleagues use their
workload as an excuse, and that the importance of self-improvement programmes
should be realized and internalized. They feel that academics should make time for
priorities and matters of importance. There was also the realization that the cycle is
initially time consuming, but subsequently it will become easier once the culture and
routine has been established.
Some colleagues still see quality assurance, and specifically self-evaluation as a
bureaucratic action, and an inhibiter of academic freedom. They realize that for lots
of people "academia is a refuge" and safe haven. Again, some of the respondents felt
that academic freedom is often misused as an excuse. Academics often felt they have
certain personality characteristics and foster a philosophy to achieve high standards.
They also claimed that they should have the knowledge, skills and attitudes to adapt
to changing environments.
The outcome of such a self-evaluation consists of a report as product. The process has
to stop at a certain point in time, otherwise it can continue without an end-point. The
consequences and results of this report are of crucial importance for the success of
the exercise. According to a participant, it should not be an "action the department
has to go through just to put a document on the table, to get the job done, to
continue with their normal processes." The questions and concerns regarding the
report that were raised by the participants, was in the first place, whether the
advantages and value of producing such a report, would filter through to everyone.
The main concern was that the departmental reports would culminate in a faculty
report, which eventually would be used in an institutional report. According to staff,
certain aspects would be "watered down," and meaningless and the whole effort
would result in a mere paper exercise. Their second concern was that the report
would not be effectively applied and utilized in the institution.
Staff stressed that recommendations stemming from the report, should be
implemented to change and improve the current situation. This does not necessarily
only mean financial investment, certain changes can be made without cost. Weusthof
37
(1994:216) refers to "passive or active utilization," of results. "Passive utilization"
refers to no immediate action being taken or merely the making of recommendations.
"Active utilization," on the other hand, refers to actively acting on the available
report, operationalized as taking measures and monitoring the implementation
thereof. Weusthof (1994:216) comes to the interesting conclusion, (which also links to
one of the previous findings about the importance of contextual influences), namely
that "in those faculties (departments) where self-evaluation procedures have been
started in response to external developments, in general the results are used passively
at both the central and decentralized level. When these processes are initiated in
response to internal conditions, however, the majority of the faculties show both
passive and active use of the results at both levels. This conclusion implies that
external incentives can be perceived as a necessary but not a sufficient condition for
successful internal quality assurance ... within faculties (departments)" (Weusthof,
1994:222).
7.4 Provision of an evaluation design
The consultant explained (Vroeijenstijn, 2001a:3) that the self-evaluation manual that
was developed should be seen as an instrument which the department could use to
ensure and improve the quality of the programmes they provide. It must be utilized to
identify strengths, and not only to point out weaknesses. The instrument provides
flexibility serving only as a guideline and allowing it to be adapted to individual
departmental needs. It should not be seen merely as a questionnaire where questions
should be answered. The questions need to be open-ended stimulating original
responses on a variety of issues.
It is important that the department must experience a feeling of ownership regarding
the development and refinement of the instrument. The development requires "close
coordination between all members" (Pekar, 1995:57). Dr Vroeijenstijn (2001a:3) used
the example of a lost person having two options, wasting time wandering around, or
buying and using a map to navigate the route to the destination. The instrument must
be seen as a map to guide the department through the self-evaluation. It helps to
direct thinking processes, but provides the freedom to formulate and determine
individual goals and aims. The aim of the manual is to structure the exercise. It was
38
interesting to note that one of the participants was of the opinion that most staff
members who received the manual, did not read it and simply put it aside. Only when
some kind of input was required of them, did they really made an effort to interact
with the document.
The respondents all felt that the instrument that was provided in the form of a
"manual" was suitable, especially for undergraduate programmes. For postgraduate
programmes, it would need to be adapted, to a certain extent. They described it as a
broad document and they "had to ask (them)selves in how much detail (they) think
(they) should go into it to get valuable results." They found it to be useful, because it
"forces you to think more widely". They all indicated that they did not "religiously
followed the model," but used it as a guideline. This correlates with the
recommendations made by Dr Vroeijenstijn (2001a:3).
8. RECOMMENDATIONS
In light of the findings of the study, it is suggested that the following
recommendations could be implemented to ensure a more effective process and to
create a more positive experience for self-evaluation. Furthermore, it would also be
meaningful for quality assurance managers within higher education engaging in self-
evaluation to improve the existing process.
➢ The goals, purpose and outcomes of self-evaluation should be determined
and communicated to all involved parties.
Mechanisms should be put in place to ensure that the goals, purpose and outcomes of
self-evaluation are communicated to all stakeholders involved in the process.
Departmental sessions should be held to provide the opportunity for reaction and input
from all the members involved. The entire department should also participate in the
development and refinement of the instrument, so that they will have ownership and
feel part of the process. A further reason for emphasizing the importance of
involvement is to establish an environment of working together towards a set of
common goals and to get total commitment from all participants (Pekar, 1995:24).
Allowances must be made for disagreements between members of a department, but,
39
ultimately, they must strive to reach a consensus between the different points of
view.
Kells (1995:16) supports the above recommendation by saying that we must
understand and clarify the goals, purpose and outcomes of self-evaluation. He argues
that it is possible that there can be numerous reasons to conduct self-evaluation, but
that the clarification thereof is a "significant matter." In the RAU context, Dr
Vroeijenstijn (2001b:5) maintains that the aim of this self-evaluation exercise was not
intended to close down any department, but to analyze strengths and weaknesses with
the aim of improving the department's functioning. In some instances, self-evaluation
is aimed at accountability, but then it should be communicated clearly to the involved
parties (Pekar, 1995:viii). The ultimate value of the exercise and the expectations
should be clearly explained.
)=. Mechanisms should be put in place to address issues and recommendations
stemming from the self-evaluation report, as well as follow-up procedures
and activities.
Issues raised during the self-evaluation should be addressed to ensure future
participation and success. The process should not just be, or perceived to be, a paper
exercise with no concrete results. Management's role in the reaction to the results
should be evident in addressing concerns and recommendations. Finances and
infrastructure should be provided where necessary, even though as mentioned
previously, some recommendations can be addressed without financial aid, but by
means of changing attitudes and thinking processes. Management (hereby not only
referring to top management, but also line functionaries) should make it possible and
motivate staff to participate. Kells (1995:34) postulates as a prerequisite "adequate
levels of interest on the part of the leaders of the institution" who should stress that it
carries a high priority.
Follow-up procedures should be clearly explained and outlined. The self-evaluation
should serve as preparation for external peer reviews. The external assessment should
be seen as an instrument to check the internal quality as reported upon in the self-
evaluation. Departments should not however only do the internal evaluation as
40
preparation for the external, but it should be a valuable exercise in itself. Kells
(1995:138) is of the opinion that "a major use of the self-study report is as a basis for
the peer review." External evaluation or peer reviews are a completely different "ball
game" with their own challenges (see suggestions for future research).
The completion of the self-evaluation should also result in the formulation of a
strategic plan based on the findings of the exercise. The strategic plan "is a
thoroughly documented set of aims for quality definition, quality assurance and
quality improvement together with a thoroughly documented set of quality assurance
procedures" (Jacobs as cited in Lategan et al. 1997:163). This plan can be updated
and changed annually. Weaknesses should be prioritized to determine which are most
important and should be addressed immediately. The format and style should be well-
organized, concise and focused on key issues. It should provide an honest and unbiased
picture and be suitable for a variety of practices. Kells (1995:95) emphasizes the
importance of the usefulness of the report. The unit administering the self-evaluation
process should play a rote in diminishing the frustration of participants and providing
guidelines and proposed formats for the report.
The self-evaluation should be combined with other, similar initiatives. Other
initiatives may include the annual report or accreditation by an external or
professional body. Management should rethink the role of the annual reports, and how
they can be incorporated in the self-evaluation. The self-evaluation should be
absorbed and integrated into other activities, and not be an additional burden or
administrative exercise.
> Internal and external contextual factors should be taken into account when
planning and conducting self-evaluation.
It is evident from the findings that certain internal and external contextual factors
influence the success of self-evaluation exercises. Internally, integrated, reliable,
regularly updated data should be available to enhance the accuracy of the self-
evaluation. Access to data and necessary information is a major cause of frustration. It
is recommended that a template of information needs to be designed, so that it is
clear what is required. Faculties and departments may choose to manage their own,
41
internal databases. Consideration should be given to the views of participants, an
aspect which is often neglected. Kells (1995:74) however reminds us that this source
of information should not become the only focus area and that other initiatives and
ways of gathering data should be explored.
Furthermore, best practices should be shared amongst departments and faculties. The
sharing of templates, formats, instruments and best practices amongst similar
departments and even among institutions are strongly recommended. Kells (1995:8) is
of the opinion that "acting together can protect individual institutions, can put
pressure on them to act on needed improvements, and can detect potentially
destructive incursions by government."
In terms of external factors, the requirements of external bodies should be integrated
and acknowledged in the self-evaluation process. Any external requirements should be
incorporated in the self-evaluation initiative. Departments exposed to such activities
should, however, be critical about the requirements and standards of such councils. If
faculties or departments already have some systems in place, the self-evaluation
exercise should be incorporated into existing practices. The importance of a healthy
relationship with external bodies cannot be stressed enough. Dr Vroeijenstijn
commented (2001b:7) that producing material required by external bodies is not
always total self-evaluation, even though it is often perceived as such. There should
always be the opportunity to reflect and make recommendations for future actions.
➢ A culture of evaluation for continuous improvement should be created.
By being dedicated to continuous improvement and striving for excellence, institutions
will constantly improve their internal performance, customer service and quality
(Pekar, 1995:21). A climate must be established and structures need to be formalized
to manage the process of evaluation before government formally requires it. However,
staff motivation is often problematic (especially during the implementation phase) and
there is a need to create a positive atmosphere. They should be motivated to identify
and rectify problem areas and to take appropriate actions. It should be clear to staff
members that the goal is also to improve their own working conditions and levels of
satisfaction. The value of such a culture should not be underestimated. Dr
42
Vroeijenstijn reminds us of "small quality" e.g. being on time, preparation for
lectures, quality of printing material etc. Such practices should be an "accepted part
of professional life" (Kells, 1995:7).
This culture should be established at faculty and departmental level by establishing
internal quality assurance systems. This view is supported by Jacobs (as cited in
Strydom et al. 1997:161, 165) who proposes that institutions should be committed to
conducting self-evaluation and other quality related initiatives at departmental and
faculty level. Heads of Departments and members of faculty need to have knowledge
of and training in quality assurance. Kells and Kirkwood (as cited in Kells, 1995:7)
support this view by providing evidence that there is a direct correlation between the
knowledge and attitudes of leaders about evaluation and the level of perceived
success after the self-evaluation process. This would also assist with creating a culture
of quality and continuous improvement. Different forms of evaluation should be
instituted in departments, such as, evaluation of study guides and teaching, practices
determining satisfaction of all stakeholders and so on. Differences within faculties
regarding thought processes should be recognized and dealt with.
> A model or guideline should be provided, but with the option for adaption
and development.
There is no one tailor made instrument that can be used by all institutions, faculties or
departments. As mentioned previously, the involved parties should have a feeling of
ownership and develop it according to the specific environment. A guideline however
is helpful and necessary. It reduces frustration and serves as a starting point. A generic
format or template could be provided, where possible, with the added advantage of
easing standardization for comparisons and analysis.
> The establishment or formalizing the role and responsibilities of a unit for
quality assurance, to steer the self-evaluation process.
The level of support and involvement provided by a quality assurance unit differs from
department to department. Those who are used to professional body accreditation, for
example, often need a different level of support. The role of the unit includes
43
facilitating and monitoring the process and motivating staff. It should also provide
training were necessary. Jacobs (as cited in Strydom, et al. 1997:169) is of the opinion
that training must be categorized to provide for different levels of staff that are
involved in the process. According to Dr Vroeijenstijn, the unit should organize
training sessions and provide support as far as possible or necessary, but should have
no say in the evaluation itself. The unit should evaluate and analyze their own
functioning and activities.
It is recommended that a self-evaluation exercise should be conducted according to
project planning principles. These principles include elements such as formulating a
realistic project plan, with deadlines, and having a project manager or leader. The
responsibility should not however reside only with the leader. He or she should be able
to delegate and assign tasks and focus areas to specific people. The establishment of a
project team or group is, according to Kells (1995:32-33), the first step in the process
of self-evaluation. This group should diagnose the situation and suggest a plan of
action. The members need to know the institution and the departmental structures
well. "The quality of the group's analysis, report, and suggestions will depend on the
leadership provided" (Kells, 1995:93). If possible, an investment should be made in
human resources, to assist with the administrative burden. The person/s appointed
should have technical expertise, not only about the methods and techniques used, but
also in terms of computer skills and logistical arrangements, to simplify the process.
9. SUGGESTIONS FOR FUTURE RESEARCH
The study exposed a number of aspects regarding self-evaluation and related activities
for suggested future research. These include:
the role, influence and involvement of management in the self-evaluation
process;
the debate regarding the administrative versus the academic responsibilities of
academic staff during self-evaluation, and how it can be addressed to enhance
productivity;
the self-evaluation was conducted at support / non-lecturing departments and
units parallel to the academic departments and their situation produces
44
numerous research possibilities such as the development of a manual that can
be adapted within their unique context;
research regarding the format, consequences and follow-up of self-evaluation
reports;
the utilization of data and the investigation of additional sources of data for
the purpose of evidence collection for self-evaluation;
practical strategies to create or enhancing a culture for self-regulation;
the role and function of a self-evaluation committee or team and the
leadership issues involved;
the process of preparing for external evaluations / peer reviews.
It is clear that, especially in the South African context, there are ample research
opportunities still to be investigated and compared with international tendencies
regarding self-evaluation.
10. CONCLUSION
This research essay focused on answering the following research question: How did
lecturers at the RAU experience the quality assurance self-evaluation process? Data
collection was done by means of in-depth interviews with lecturers, and to a lesser
extent participant observation. After analyzing the data, it was found that there are a
variety of responses regarding the self-evaluation process, not all of which
corresponded with my initial assumptions. Most of the lecturers realize the value and
need of such an exercise, but there are certain definite internal and external
influences that can hinder the process. Because they also experience it as a time-
consuming exercise, they appreciate the provision of a structured manual that serves
as a guideline. They do however want the freedom to adapt and organize the exercise
according to their individual needs.
It is evident from the study that self-evaluation should be presented in a positive light
to make academic staff realize that is does not only add value to their personal work
performance, but also has a ripple effect within the broader institution. It results in
improved standards within the department, and ultimately the institution. Thus, the
improvement will be evident, firstly, on an individual level. The individual will be able
45
to present higher quality programmes, in terms of content and coordination with other
programmes. This increased confidence of academic staff members will influence the
functioning on departmental level. Departments will be efficient and optimally
utilizing resources, both human and otherwise. The benefits of the constant analysis
and refinement of programmes, range from identifying and correcting weaknesses, to
being in synch with market demands. Lastly, this will have a ripple effect on the high
standards offered by an institution. The institution would have a reputation as being
committed to continuous improvement. This is reflected in the following diagram:
Diagram 10.1: Ripple effect of self-evaluation:
High standards on institutional level
Effective departmental functioning
..s........„ Improved individual
performance
This study included a review of some of the available literature on the topic of self-
evaluation, it explained the processes of data collection and analysis, and also
suggested some possible recommendations for consideration by departments
undertaking the same exercise. Future research into the area of quality delivery at
tertiary institutions and the implementation of well-planned self-evaluation exercises
will ensure that we maintain the academic standards and values required by the
national and international community.
46
Eating an elephant
Why on earth do we have to do this? I just don't have the time! I am an academic. You're imposing on research time.
Strengths Weaknesses Opportunities Threats Just let me be...
Give me a format! Where's that manual? What wilt they do with this? Only now I realize why.
Some say it's like riding a wave Highs lows highs lows Or how do you eat an elephant You go about it bit by bit.
Nicolene Murdoch
(Composed at a Qualitative Research Methodology workshop while exploring alternative methods to data analysis)
LIST OF REFERENCES
Bailey, A.C. (1996). A Guide to Field Research. Thousand Oakes: Pine Forge Press.
Barlosky, M. ft Lawton, S. (1995). Developing Quality Schools. Toronto: Kodak Canada
Inc and the Ontario Institute for Studies in Education.
Barnett, R. (1992). Improving Higher Education: Total Quality Care. The Society for
Research into Higher Education ft The Open University.
Biggs, J.B. (1991). Teaching for Learning: the View from Cognitive Psychology.
Hawthorn: ACER.
Brennan, J., Frazer, M. ft Williams, R. (1995). Guidelines on Self-evaluation. London:
Open University Validation Services.
Brink, J.A. (1997). Quality Promotion Unit: Quality Audit Manual. Pretoria: SAUVCA.
Byrne, M.M. (2001). Evaluating the Findings of Qualitative Research. AORN Online:
Research Corner. Available from: http://www.aorn.orq/joumal/2001/marrc.htm
(Accessed on 3 October 2001).
Department of Education. (1997). Education White Paper 3: A Programme for Higher
Education Transformation. Pretoria: Department of Education.
De Vos, A.S. (Editor). (1998). Research at Grass Roots: A Primer for the Caring
Professions. Pretoria: Van Schaik.
Fourie, M. (2000). Self-evaluation and external quality control at South African
universities: quo vadis? Paper presented at the 22 nd annual EAIR Forum, Berlin.
Frazer, M. (1992). Quality Assurance in Higher Education. In Craft, A., Quality
Assurance in Higher Education: Proceedings of an International Conference, Hong
Kong, 1991. London: The Falmer Press.
47
Guba, E.G. ft Lincoln, Y.S. (1989). Fourth Generation Evaluation. Newbury Park,
London: Sage Publications.
Harvey, L. (1995). (Editorial). Quality in Higher Education, 1(1):8-12.
Higher Education Quality Committee (HEQC). (2001). Founding Document. Pretoria:
Council on Higher Education.
Hoepfl, M.C. (1997). Choosing Qualitative Research: A Primer for Technology
Education Researchers. Journal of Technology Education, 9(1):1-17. Available from:
http://scholar.lib.vt.edu/ejournals/JTE/v9n1/hoepfl.html (Accessed 5 October 2001).
Kells, H.R. (1995). Self-study Processes: A Guide to Self-evaluation in Higher
Education. Washington, D.C.: ORYX Press.
Kimball, E. (2000). Departmental Reviews Place Emphasis on Self-evaluation.
Heraldsphere News. Brown University. Available from:
http://www.browndailyherald.com/stories.cfm?S=2fID=943 (Accessed 31 May 2001).
Lewis, R.G. a Smith, D.H. (1994). Total Quality in Higher Education. Florida: St. Lucie
Press.
Loder, C.P.J. (1990). (Editor). Quality Assurance and Accountability in Higher
Education. London: Kogan Page.
Maykut, P. a Morehouse, R. (1994). Beginning Qualitative Research. A Philosophic and
Practical Guide. London, Washington, DC: The Falmer Press.
Merriam, S.B. (1998). Qualitative Research and Case Study Applications in Education.
San Francisco: Jossey-Bass Publishers.
Miles, M.B. a Huberman, A.M. (1994). Qualitative Data Analysis. Second Edition.
London: SAGE Publications.
48
Myers, M.D. (2001). Qualitative Research in Information Systems. Available from
http://www.misq.org/misqd961/isworld/ (Accessed on 24 October 2001).
Nachmias, C. (1997). Social Statistics for a Diverse Society. California: Pine Forge
Press.
National Commission on Higher Education. (1996). A Framework for Transformation.
Pretoria: Department of Education.
New Jersey Commission on Higher Education. (2000). Accountability in Higher
Education. The 4th Annual System wide Report. New Jersey Commission on Higher
Education. Available from: http: / /www. state. n j. us/highereducation / reports. htm
(Accessed 2 June 2001).
Open University. (1999). Self-evaluation in Higher Education: The Guide. Centre for
Higher Education Research and Information. United Kingdom: Hobbs the Printers, Ltd.
Patton, M.Q. (1987). How to use Qualitative Methods in Evaluation. London: SAGE
Publication.
Patton, M.Q. (1990). Qualitative Evaluation and Research Methods. Second Edition.
London: Sage Publications.
Pekar, J.P. (1995). Total Quality Management: Guiding Principles for Application.
Philadelphia: American society for testing and materials.
Sagor, R. a Barnett, B.G. (1994). The TQ• Principal: a Transformed Leader. Thousand
Oaks: Corwin Press.
Sallis, E. (1996). Total Quality Management in Education. London: Kogan Page.
Schon, D.A. (1987). Educating the Reflective Practitioner. San Francisco: Jossey-Bass.
49
Silverman, D. (2000). Doing Qualitative Research: A Practical Handbook. London:
SAGE.
Strydom, A.H., Lategan, L.O.K. Et Muller, A. (1997). Enhancing Institutional Self-
evaluation and Quality in South African Higher Education: National and International
Perspectives. Bloemfontein: The University of the Free State.
Trow, M. (1996). On the Accountability in Higher Education in the United States. The
Princeton Conference on Higher Education, March 1996.
Vroeijenstijn, A.I. (1990). Autonomy and assurance of quality: two sides of one coin.
The case of quality assessment in Dutch universities. Higher Education Research and
Development, 9 (1 ): 21 -36.
Vroeijenstijn, A.I. (1995). Improvement and Accountability: Navigating between Scylla
and Charybdis. Guide for External Quality Assessment in Higher Education. London:
Jessica Kingsley Publishers.
Vroeijenstijn, A.I. (2001a). Fieldnotes - Visit to RAU 26 February 2001 - 3 March 2001.
Vroeijenstijn, A.I. (2001b). Fieldnotes - Visit to RAU 11 June 2001 - 14 June 2001.
Wellman, J.V. (2001). Assessing State Accountability Systems. Change, March / April
2001:46-52.
Weusthof, P.J.M. (1994). De Interne Kwaliteitszorg in het Wetenschappelijk
Onderwijs: Een Onderzoek naar de Kenmerken en het Gebruik van Zelfevaluatie aan
Nederlandse Faculteiten. Enschede: CSHOB.
Woodhouse, D. (1995). Audit Manual: Handbook for institutions and Members of Audit
Panels. Wellington: New Zealand Universities Academic Audit Unit.
50
1
ADDENDUM 1 ID 1
N — Ek sal prof net so min of meer vertel waaroor gaan die studie. Dit gaan oor die self-
evaluering wat ons nou hier by RAU doen, maar dit gaan oor ervarings daarvan sodat ons
in die toekoms negatiewe gevoelens kan uitskakel en positiewe gevoelens op die
voorgrond bring sodat die proses makliker sal wees. Ander instellings is ook
genteresseerd in die studie, omdat almal nou besig is om te begin met hierdie oefening
en omdat dit nog baie onseker is op hierdie stadium. Die HEQC (Higher Education
Quality Committee) het ook aangedui dat hulle baie sterk gaan leun op self-
evalueringsverslae van instelling, so dit beteken maar net dit is nou nie meer 'n keuse nie,
dit is nou 'n vereiste, so, maar die feit bly staan dat daar is maar redelik gevoelens
daarrondom. So, die plan met die studie is om nou 'n kort onderhoud te voer en weer
bietjie later in die proses warmer julle meer nader aan inhandiging is. Ek kan nie wag tot
die verslae eers in is nie, want ek moet die einde van die jaar ingee, maar weer 'n bietjie
later, wanneer dit nou meer gefinaliseer is. Net nou 'n kort onderhoud en dan later as prof
nie omgee nie. So wat ek wil weet is maar net in die algemeen, hoe ervaar prof self-
evaluering, wat dink prof daaroor, enige gevoelens daaroor? Dink prof dis nodig?
I — [stilte] Goed, uh, [stilte] ek dink dis beslis nodig. [stilte] Um, veral wat betref die
nagraadse opleiding, en daarmee bedoel ek die magister en doktor studie [stilte] was daar
in die verlede nie baie riglyne neergele, enersyds deur die department en of fakulteit nie.
So persone wat nuut ingekom het in die sisteem is maar aan hulleself oorgelaat om 'n
magister of doktorstudent op te lei. Nou, as 'n mens begin kyk na kriteria wat geld vir
goeie opleiding en jy begin mooi clink oor al die fasette betrokke by die opleiding, dan
kom jy agter dat 'n mens eintlik maar [stilte] 'n student inneem en hoop dat by die
vaardighede gaan aanleer oor die verloop van die tyd. Daar is nie 'n doelbewuste
evaluering van die student se vordering nie. Goed, daarmee bedoel ek nie die vordering
met navorsing nie, maar ook vordering in vaardighede, daardie uitkomstes van SAQA. So
dit was vir my nogal, net weer onder besef gebring hoe belangrik dit is dat mens weer
dink oor wat jy met jou opleiding wil bereik en tot hoe 'n mate 'n mens dan suksesvol is.
2
Baie keer gebeur dit dan dat studente dan goed vorder met een aspek van die navorsing,
maar dat daai ander vaardighede agterwee bly.
N — Dit word nie noodwendig getvalueer op 'n stadium nie.
I — Dit word nooit geevalueer nie. In ons departement het ons net die verhandeling of die
proefskrif wat geeksamineer word. Die ander fasette van eksaminering is baie subjektief.
Ons kyk na 'n student se houding, ons kyk na die aanbieding van seminare, maar daar
word nooit werklik 'n punt aan toegeken nie. Vroeer fare was daar nog 'n sisteem by ons,
afgesien van die verhandeling of die proefskrif, ook 'n mondelinge eksamen gehad het.
Dit was baie goed gewees. Daar kon jy die student se vakkennis toets, jy kon sy vermoe
om sy kennis en vaardighede oor te dra en te deel kon jy evalueer. Jy kon sy diepte peil.
N — Ja, weier as net dit wat by inhandig.
I — Baie keer gebeur dit dat dit wat inhandig word is 'n produk van die student sowel as
die studieleier. Dit is nie net die student se eie nie.
N — Hulle het baie leiding nodig.
I— So, swak studente en goeie studente [stilte] laat ek so se: 'n swak student kan 'n beter
punt kry as 'n goeie student, omdat die studieleier hom gedruk het.
N — Ja, op die ou einde gaan almal deur die sisteem.
I — Almal kry dieselfde graad, maar party [stilte] kom beter deur die sisteem in terme van
die uitkomstes. So dit was vir my sinvol om deur die hele oefening te gaan. My aandeel
in die departement se evaluering is die navorsing. Ons het reeds met voorgraadse tot en
met honneurs situasie-analise gedoen, ons het die SWOT-analise gedoen. Selfs daar
[stilte] was duidelik areas waarop ons baie kan verbeter.
3
N — Wat mens nie altyd aan dink of aandag aan gee nie.
I — Ja, ja. Veral wat betref die onderlinge samehang van die kursusinhoude. Urn [stilte] In
ons opset doseer een dosent gewoonlik een voile semester kursus. En dit is dan
gewoonlik 'n spesifieke onder afdeling van die vak. So [stilte] hoe daardie onder afdeling
aangebied word, en hoe daai raakvlakke met die ander aspekte van die vak aangetoon
word vir die student, kan mens nooit na waarde skat nie. So, ek dink een van die dinge
wat ons beslis in die toekoms sal na moet kyk is beter koordinering tussen die
verskillende modules.
N- Ja, want dis dikwels een dosent wat een semester doen en wat baie goeie werk doen,
maar wat nog steeds nie saamhang met die ander kursusse nie.
I — Ja, en dan aan die einde van die dag as die student daardie geheelbeeld oor die vak
moet kry , dan is die onderbou nie so sterk nie.
N — Hy sien nie die groter prentjie nie. Ja, waar dit inpas.
I — Urn [stilte], goed so die noodsaaklikheid vir so self-evaluering dink ek is daar. Die
voordele wat uit so 'n self-evaluering kan kom is baie goed dink ek. [stilte] Hoe ervaar ek
dit? Dit is tydrowend, dis frustrerend. Goed, ons moes nou van nuuts of begin om 'n
databasis saam to stel. Daar was klein gefragmenteerde databasisses, maar niks
oorkoepelend nie, niks volledig nie. So dit was een faset wat ons ook na moes kyk.
N — Maar prof, dit is nou half opgestel, en sal in die toekoms weer kan gebruik word. Dit
is die tipe van ding wat jy een keer doen, dan kan hy aangepas word.
I — Ons kan nou op 'n jaarlikse basis net aanvul. Dan die samewerking van die kollegas is
ook 'n probleem. In hierdie dokument van Dr Vroeijenstijn se hy dit moet 'n
gemeenskaplike poging wees, en dit is baie waar want, [stilte] anders [stilte] die persoon
wat dit opstel of die persone wat dit opstel ervaar die frustrasies van gebrek aan
4
samewerking en die voordele verbonde aan die evaluerings vloei nie oor na die groter
department toe nie.
N — Dink prof dit is die bewustheid van die kemgroepie wat dit hardloop, besef die
belangrikheid of loop 'n pad met die projek, maar die wat op die kantlyn half ingetrek
moet word, besef nie altyd die waarde of belangrikheid.
I — Dit is baie waar. En ek dink dit is uiters noodsaaklik, want dit help nie dat [stilte]
enkele individue verryking kry uit die evaluering nie, dit moet die department wees, want
dit is tog waaroor dit gaan. So, [stilte] dit het nou ongelukkig saamgeval met ons NRF-
aansoeke wat einde van hierdie maand moet in wees, so die personeel is baie besig en dit
bring frustrasies mee, maar ek kan se, urn, hulle kan beter samewerking gee. Mens kan
hulle nie te veel druk nie, want die klasgee en navorsing moet prioriteit geniet, daar is
vraestelle wat nagesien moet word.
N — Hulle het maar hulle ander pligte.
I — Ja, so ek [stilte] een gevaar wat ek sien in hierdie hele oefening is dat dit nie wyere
verryking gaan meebring nie. Dat dit net 'n aksie is wat die departement deurgaan om 'n
dokument op die tafel te le en die werk klaar te kry, sodat hulle met hulle normale ou
prosesse kan aangaan.
N — Laat dat dit nie regtig gaan infilter nie, die voordeel daarvan gaan deurfilter nie.
I — Dit sien ek as 'n moontlike gevaar, so, maar ons beplan om 'n departementele beraad
te hou, binnekort, waarop die hele departement dan insette gaan lewer oor die finale
dokument. Maar hierdie berade het ook maar hulle beperkte gevolge. [stilte] Wat was nog
vrae gewees?
N — Ek wil net you aanhak by iets wat prof nou gese het. Die departement was een van
die min departemente wat die hele departement bygetrek het by die aanvanklike besoek
5
van dr Vroeijenstijn. Die ander departemente het net die kernkomitee of party net die
voorsitter bygetrek. Maar dink prof dit het 'n invloed gehad, het (Iasi sessie nie gehelp
dalk om hulle, of was hulle nogsteeds half buite op die kantlyn gewees al het hulle, dis
asof hulle nog steeds nie die ems van die saak... ek het gedink dit sal anders wees hier
omdat hulle by was toe by hier was.
I— Dit is moeilik om te veralgemeen want ons is 'n baie diverse departement. In A het
hulle vier onderafdelings, en dan het ons B as 'n parallele department en van die, ek sou
se 80% van die personeel het dit bygewoon, die inligtingsessie, maar in daai stadium het
hulle nie werklik besef dat daar &ens in die toekoms insette van hulle verwag gaan word
nie. Hulle was meer passiewe ...
N — observeerders.
I — Ja, [stilte] goed, hierdie dokument is uitgedeel, en hulle het dit gekry en neergesit. Ek
is seker dat afgesien van die mense wat die aksies loods, die ander nog nie die moeite
gedoen het om werklik dit deur te gaan nie.
N — Ja, daar is nie iets konkreet, dadelik van hulle gevra nie.
I — Ja, so [stilte] maar ek moet ook se, dit is maar hoe dit met verskeie aksies hier gaan by
die Universiteit. Die personeel is werklik oorlaai sodat enige bykomende aktiwiteit word
nie ... word as ekstra werk beskou en daar is half 'n gevoel daarteen. 'n Weerstand.
N — Dis net nog iets wat bygekom het.
I — Wat net nog tyd wegneem van navorsing, die kernbesigheid.
N — Ons ervaar dit venal by die sentrum, ervaar ons dit omdat ons "skills development'
en daai tipe goed wat van die regering of kom en goed vereis, ervaar ons dat ons as
boodskappers altyd geskiet word. Want soos hierdie kom ook, dit is ook iets wat, soos
6
"skills development", die mense haat die vorms, hulle sien dit net as nog 'n ding, hulle is
vraelys-uitgeput op hierdie stadium. Maar ek weet nie of mense nie besef dat dit goed is
wat ons moet doen, hierdie is iets wat ons moet doen ek clink as ons die bewustheid in die
Universiteit kan vergroot. Ons wil, ons wil net he die HEQC moet konkrete riglyne op
die tafel sit, en se dit is wat hulle gaan vra, en as hulle nie dit het nie. Ons kan half lees al
wat gaan kom, soos hierdie, hulk het dit vasgemaak. Hulle kan nie by elke instelling, by
elke departement uitkom om 'n eksteme oudit to doen nie. Hulle gaan hierdie verslae vat,
wat saamgevat gaan word in 'n institusionele oudit en dit is wat hulle RAU op gaan
evalueer of assesseer.
I — Wat ek oor gewonder het, elke departement moet nou so 'n dokument gaan opstel, so
dan gaan sekerlik per fakulteit 'n dokument opgestel word. Gaan daar nie in die proses
baie van die aspekte wat na yore gekom het verwater of verdof nie. Dat so universitere
dokument dan eintlik baie niks seggend gaan wees.
N — Ons wil ook waak danrteen dat dit 'n baie algemene, sso prof nou se, niksseggende
papieroefening gaan wees. Dit is die grootste gevaar van self-evaluering is dat daar op die
ou end 'n dokument op die tafel is, en dat dit nie gaan voordeel inhou vir die hele
departement of die hele Universiteit dan nie. Maar die HEQC het gese hulle gaan
departementele verslae, hulle gaan nie net institusioneel doen nie...
Onderbreking
N — So ons hoop dat dit nie so gaan wees nie, en dat hulle duidelik riglyne en ... maar
hulle weet self nog nie hoe dit gaan wees nie, dit is die probleem. Ons wens dat hulle
hulleself net wil uitsorteer. Ons woon al hulle forums en berade woon ons by, want ons
probeer agterkom wat wil hulle he, wat gaan hulle doen. En hulle is baie duidelik dit gaan
nie 'n papier oefening wees nie, en daar gaan definitief... En nou is daar soveel ander
dinge soos die samewerking tussen die instellings met C en die D wat nou weer 'n
invloed het.
7
I — Ek moet se in van my gesprekke met van die personeel het dit ook na yore gekom dat
ons gaan nou deur hierdie oefening maar hoe relevant is dit nou gesien teen die ander
veranderings...
N — Wat hulle ook mee besig is. Ons het nou met C en D, nie net met C en D, daar was 'n
kwaliteitspersoon van elke instelling en die HEQC het gese spesifiek waar "mergers"
aanbeveel is, hulle van self-evalueringsverslae gebruik maak om te besluit. Maar dit
maak dit moeilik want ons wil he die doel van self-evaluering, dis mos of aan die een
kant "accountability" of "improvement". En "accountability" is "ons kan julle toemaak
op grond van dit". Ons wil nie he dat dit, dat departemente met daardie vrees loop hulle
gaan toegemaak word nie op grond van wat hulle daarin sit, want dan gaan hulle nie
eerlik wees nie. Dit moet vir ons meer 'n "improvement" tipe van oefening wees. Hulle
het wel gese dat, en hulle se dit deur en deur, hulle gaan swaar leun op self-
evalueringsverslae. So dit wys maar net vir ons, ons is op die regte pad en hulle is baie
geinteresseerd wat RAU doen, omdat ons een van die eerstes... die technikons is noual
heelwat verder in die proses van SERTEC en so, maar in terme van universiteite. Daar is
self-evaluering in plek by van die ... maar dit is nie regte self-evaluering nie.
I— So twee jaar terug was ek een van twee van die komitee wat die E department van F
Universiteit moes evalueer. [stilte] Dit was ook self-evaluering van hulle kant af, maar
glad nie geskoei op hierdie model nie. Dit wat hulle aan ons voorgele het was maar
voorbeelde van hoe hulle dit doen, nie dat hulle hulleself die vraag afgevra het: "is dit die
regte manier om dit te doen?" of "kan ons dit dalk beter doen nie?". Dit is aan ons
voorgele, ek is seker die negatiewe aspekte is weggehou van ons af.
N — Dit is so maklik om dit te doen.
I— Op die oog af was alles goed gewees, dit was anders as RAU sin, en dit was anders
die Universiteit van G sin, maar ek kon nie werklik kritiek fewer daarop nie. Self-kritiek
is eintlik goed. Um, ek het so vraelys opgestel voor ek begin werk het aan die
navorsingstruktuur en ek het vrae gevra soos: vind die personeel die navorsing in die
8
department bevredigend? Aan die een kant het almal van hulle gese ja, navorsing is uiters
krities vir hulle, en dat jy nie werklik in die H fakulteit 'n dosent kan wees sonder om ook
`n navorser te wees nie. Toe was ek baie verbaas om te sien hoe negatief die personeel
navorsing eintlik ervaar, en dit is dan veral die senior personeel. Die jong mense wat nou
nog besig is met hulle eie studies en so aan, hulle ken nog nie sisteem so goed nie en
hulle is nog half meer optimisties, maar die meer senior mense in die departement was
feitlik deur die bank baie baie negatief in hulle ervaring van navorsing. So aan die een
kant is dit 'n aspek wat almal se "ja, dit is uiters krities vir my as dosent in my
werksbevrediging", maar aan die ander kant ervaar hulle dit baie negatief Maar dit is nou
die frustrasies wat deur burokrasie, befondsing, gebrek aan infrastruktuur, gebrek aan
aparatuur meegebring word. So ek dink dit is 'n [stilte] situasie wat aangespreek sal moet
word, want jy kan tog nie vir so belangrike been van jou werkskomponent heeldag net
frustrasies ervaar nie.
N — Dit gaan later die ander komponente beinvloed.
I — Dit gaan, dit gaan. Ek is seker as die finale verslag deurgaan na die Delman toe, gaan
hierdie aspek baie afgewater wees. Dit gaan nie so sterk genoem word in daai verslag nie,
want ons wil nie die dekaan ontstel nie. Hy probeer van sy kant of doen as wat by kan.
[stilte] Ja, so, ek weet nie. Aspekte van die akademiese oefening: Dit is daar maklik om te
se ons moet hier of daar verbeter en ons kan dit deurvoer en resultate verwag.
N — Jy kan dit meet en sien dit is beter.
I — Navorsing is moeiliker. Ek is juis nou besig om 'n vraelys op te stel vir die studente
oor hoe ervaar hulle, hoe bevredigend, die "satisfaction" komponent wat hulle kry uit die
opleiding uit. Daar wil ek ook nogal bietjie [stilte] diepgaande vrae vra om werklik te kan
bepaal hoe bevredigend die opleiding vir hulle is. Ek dink ook wat hulle aanbetrek, hulle
dink nie eintlik wat hulle uit die opleiding behoort te kry nie.
N — Hulle weet nie eintlik nie.
9
I — Een van die vrae byvoorbeeld is: dink jy dat die opleiding wat jy by RAU kry jou
toerus om 'n goeie pos in Suid-Afrika te kry, en die B-deel van die vraag, om 'n
moonlike post-dok oorsee te kry. Daar speel mens nou of die markaanvraag in Suid-
Afrika, teenoor die diepte opleiding wat nodig is om oorsee in 'n post-dok in te gaan.
[stilte] Miskien is ons skuldig daaraan dat ons nie genoeg aandag aan beide strome gee
nie. Dat ons miskien studente oplei of navorsers oplei met die idee dat hulle baie goeie
navorsers gaan wees, terwyl die markaanvraag in Suid-Afrika is meer vir mense met 'n
honneurs.
N — En nie sulke hoe, in diepte navorsers nie.
I — Ja, en dan ook [stilte] wat ek in die verslag wil noem is dat die uitkomste, daar moet
deurplopend evaluering wees van hierdie uitkomste. En as 'n student nie aan 'n sekere
kategorie van uitkomstes voldoen nie, dan moet die studieleier dit met horn of haar
bespreek. Urn [stilte] want weereens, ons het innames, ons het hierdie swart kassie, en
ons het studente wat anderkant uitkom. Ons weet nie wat in die swart kassie aangaan nie.
N — Ja, hulle gaan deur die proses en hulle kom anderkant uit, maar dis wat gedurende die
proses gebeur en die vaardighede, mens wil weet of hulle ander vaardighede as net bloot
die skryf van 'n navorsingsverslag.
I — Ja, dit is, die 'criteria wat ons gebruik is as daar se nou maar twee navorsingsartikels
uit 'n magister kom, is dan beter as 'n magister wat dan net een artikel. En dis is nie
noodwendig so the. [stilte] 'n Student kan tydens sy studie as 'n tegniese handleier
ontmoet word, wat eintlik verkeerd is, ofjy kan hom werklik probeer oplei as
wetenskaplike, wat eintlik die regte ding is. Ja...
N — So prof hoop dat hierdie oefening daai tipe probleme gaan aanspreek. Dat dit nie
afgewater gaan word in 'n algemene verslag nie, en weer deur die mat gaan val of ...
10
I — Ja, ek dink as ons vat, die departement se verslag nou by die fakulteit eendag uitkom.
Dan hopelik sal daar [stilte] van die fakulteit se kant 'n bewuswording wees om hierdie
aspekte van nagraadse opleiding meer formeel te maak.
N — Dat hulle sal reageer daarop.
I — Om 'n ander voorbeeld te gebruik, oorsee is daar 'n komitee wat kyk na studente. Dit
is nie net 'n enkele studieleier nie. So daardie komitee is gewoonlik dan ten minste drie
persone, senior, ervare navorsers, wat dan daardie student se vordering moniteer.
N — Dit is nie 'n subjektiewe, een persoon studieleier nie.
I — En, uh, [stilte] die komitee evalueer dan'n paar keer die student se vordering en voor
die student mag inhandig moet die komitee toestemming gee. En hier by ons is daar baie
keer druk van die student se kant af: "ek moet nou inhandig", terwyl jy is huiwerig. Is die
student werklik gereed?
N — So 'n uitvloeisel hiervan, sal hopelik 'n meer formele sisteem...
I— Dat die resultaat 'n meer formele sisteem sal wees, gemeet aan die uitkomstes wat
verwag word.
N — En hierdie model, waarin RAU ingekoop het, dink prof dit was 'n goeie idee, of dink
prof dit was 'n goeie model, of dit was te voorskrywend, of... want mens wil 'n tipe van
`n riglyn he, maar ons wil steeds nog ook probeer om akademiese vryheid toe te laat.
I — Uh, ek dink die model is goed, vir voorgraads is by baie goed, vir nagraads is daar
[stilte] kan daar bietjie strenger kriteria... Ons het in die department besluit ons gaan horn
nie streng navolg nie.
N — En dis heeltemal goed.
1 1
I — Ons gaan hom bietjie aanpas volgens ons behoeftes en waar daar dan areas is wat ons
voel die model nie honderd persent van toepassing is nie, gaan ons dit net so noem en
aangaan.
N — Die gedagte is ook om as daar dirge is wat byvoorbeeld nie in die departement is nie,
byvoorbeeld om die satisfaksie van die mark te bepaal dan noem 'n mens dit, daar is nie
op hierdie stadium so iets in plek nie, maar ons is besig om iets daaraan te doen. Want
ons kry in party departemente, ons hoor van ander instellings ook, wat vreeslik skop teen
enige vorm van 'n model, wat baie, baie sterk steun op akademiese vryheid. Die
probleem is net dan word alle aspekte miskien nie aangespreek nie.
I — Wat vir my nuttig is in hierdie model is dit dwing jou om oor 'n baie wyer gebied te
dink.
N — Goed wat mens nie noodwendig self sou aan gedink het nie.
I — Ja, ja. As jy nou hierdie matriks gaan volg en puntsgewys elke [stilte] aspek die
situasie-analise moet doen, dit onderling bespreek en dan 'n SWOT-analise dan se wat
moet verbeter, dan dink ek [stilte] raak jy aan... dan kry jy diepte. Jy raak aan aspekte
wat baie keer, jy het momentum, maar oorspoel word. Dinge [stilte] gebeur maar met 'n
rede, op daardie patroon geskiet, maar weereens, dit is nie noodwendig die gewensde
patroon nie en dit spreek baie keer nie aan wat dit behoort nie.
N — Ons boor dit baie. Hulle doen dit al vir twintig jaar, so dit moet reg wees, en dis juis,
as jy dit al vir twintig jaar doen dan is daar fout. Maar hulle skop maar... Ek moet se by
hierdie departement het on regtig 'n openheid gekry om self te kritiseer en self te
evalueer want ons sien by ander instellings ook dat die self-evaluering wat gedoen word
is bloot 'n af'tick" van 'n vraelys, daar is geen evaluering en refleksie en daai situasie-
analise wat gaan ons nou doen om dit beter te maak.
12
I — Ek dink miskien wat uh, [stilte] ons as wetenskaplikes moet baie keer dinge evalueer
en die betekenisvolheid van 'n stel data, is daar so 'n ingesteldheid het en dat ons die
dinge op gesigswaarde aanvaar nie.
N — Jo, hulle is bang om te kritiseer, bang om self te kritiseer.
I — In ons werksopset, as jy nou navorsingsresultate voorle vir publikasie, dan word dit
deur sogenaamde "peer review" geevalueer. En daardie evaluering is partykeer baie
streng en baie vernynig. So ons is gewoond daaraan om andere te kritiseer, partykeer
onsself ook. Dis nie altyd lekker om kritiek te hoor nie. Jy sien tog die meriete van die
kritiek in.
N — Mens moet dit tog net ook evalueer, die kritiek evalueer, of daar waarde daarin is, ja.
I — En baie keer is daar, en dan is daardie kritiek tot voordeel en verbetering van jou eie
navorsing. So dit is 'n kultuur waaruit ons kom, so miskien is dit hoekom ons meer oop is
vir die proses. As dit nou net nie was vir die hoe werkslading nie, dan kon ons werklik
baie meer tyd daarin gesit het.
ADDENDUM 2
DETAILED NOTES ON THE VISIT OF DR TON VROEI3ENSTL1N 26/02/2001 — 03/03/2001
MEETING 1
Date:
Monday, 26 February 2001 Time:
12:00 Venue:
CHES — conference room CHES, Chairperson: Quality Care Committee
Our expectations of this introductory visit:
This visit is mainly seen as a "head turning" exercise, to sensitise colleagues regarding the self-evaluation (SE) initiative. Some colleagues still see quality assurance / SE as a bureaucratic action, and an inhibiter of academic freedom. They need to be informed about the global and local initiatives and tendencies in this regard, to show we are in line with international practices. The faculties must be made aware of the SE initiative and some initial uncertainties needs to be addressed.
The program:
The meeting with the Quality Care Committee (QCC) is especially important because:
They must be able to feed a positive attitude back to the faculties; They need to be provided with an opportunity to raise questions and concerns, as they have been part of this journey from the beginning.
The faculty meetings are important because: They are often not informed and "converted" as yet; Some of the faculties are more problematic/negative than others and They have to buy into and support the concept of SE for this to be an effective exercise.
Non-academic departments: The manual provided is, at this stage, strictly written for academic departments. Each support unit is unique and distinctive. Some support units also have an academic focus e.g. CHES. There will be a special meeting to address their needs.
1
Dr Vroeijenstijn's expectations:
To play a facilitating role. To convey the important message that this exercise is costly and time-consuming. To prepare the staff of RAU for what to expect. To get feedback on the draft SE-manual. To identify departments to take part in pilot project of SE. To call for the election of small work groups within participating departments. To sensitise staff that this will eventually be expected by the government and Higher Education Quality Committee (HEQC), and prepare them for an external audit. To stress the importance of the training of staff who will responsible to manage the process. To keep other departments informed. To identify problem areas. To stress the importance of SE for self-improvement, and not just as part of a governmental or external initiative.
The role of the QCC and CHES:
To develop the instrument. To facilitate the implementation of the instrument within faculties. To organise the process on a central level. The responsibility lies within the faculties to implement the SE.
Foreseen problems:
The IT-system because of the insufficient provision of records and the lack of reliable data.
Preparations for Dr Vroeijenstiin's next visit in May:
Preparation of departments that volunteered to take part in the SE exercise. Dr Vroeijenstijn would train the internal work groups within the identified departments. Exploration of the appropriateness of the manual for the different contexts. Provision of feedback about the manual to Dr Vroeijenstijn. The departments must experience a feeling of ownership regarding the instrument.
2
MEETING 2
Date: Monday, 26 February 2001 14:00
Venue: Conference room — Faculty of Education and Nursing Group: A
Dr Vroeijenstijn addressed the following issues:
This is not a bureaucratic, top-down approach. We, as an institution, have to personally take responsibility for quality, in our own environment and context. The SE-manual must be seen as an instrument departments can use to ensure and improve quality. It must be used to identify strengths, not only to point out weaknesses. The manual is not to be forced upon any department, it must be seen as a guideline and the department should decide what is appropriate for their situation. The manual must be adapted to be program, department and university specific. The whole department must be involved in this Icustomisation'-process. The SE-manual will also be adapted for the non-academic departments / support units according to their specific situations and the nature of services offered. The higher education context becomes more competitive each day. Dr Vroeijenstijn used the example of a lost person having two options, wasting time wandering around, or buying and using a map to navigate the route to the other side. The instrument must be seen as a map to guide us through this evaluation. A valuable SE-exercise needs involvement and insights from external sources. Departments sometimes assume they are doing the right things, and therefore they need an external validation of this belief. But it is important that the criteria for the internal and external evaluation are comparable. SE is context-based. The criteria for evaluating a 5-star restaurant is not the same for evaluating a fast food outlet e.g. McDonalds. Experts in the field of study should participate in selecting and determining the criteria for the specific area/department. The department would therefore play an active role in determining criteria for their own SE. All the stakeholders involved have different and diverse expectations, attitudes and demands regarding SE. Benchmarking should be done to compare the institution to other, similar national and international constituencies. As we all know, quality is not static or easy to measure. Dr Vroeijenstijn used the example of wine. You can measure whether it contains all the necessary ingredients, but that does not necessarily mean it is a quality wine. The external panel of evaluators should be experts in the specific field; the ideal group is usually 5 people.
3
Students should also be involved and aware of the initiative. They should be allowed to air their views for a specific part of the evaluation. A student should also be included in the external evaluation committee. The issue of intemationalisation vs. African relevance should be considered. We should deliver students that are trained for a specific employer. The input from the employers should also be considered when determining criteria for SE. The goals and objectives of a department should be aligned with the mission and vision of the university. A synthesis must be reached between a bottom-up and top-down approach. A SE-exercise must be conducted every x number of years in a cyclical manner. The HEQC will have their own cyclical process, and RAU would have to adapt accordingly. Client satisfaction is a major part of any SE-initiative. Departments should establish what information they require from their 'clients' and they should develop their own unique instruments to gather this information. This is a pattern of thought that must be established throughout the university. The issue of centralisation vs. decentralisation is an international problem. This initiative depends on support from management, but on the other hand, they cannot enforce it, the responsibility and ownership must lie within the department.
MEETING 3
Date: Tuesday, 27 February 2001 Time: 8:00 Venue: Conference room — Faculty of Education and Nursing Group: B
Dr Vroeijenstijn gave a brief introduction to the proposed project. (See meetings 1 & 2)
The Nursing department volunteered to participate in the initial SE-exercise. They are used to accreditation from a professional body. The Health Services are quite far advanced in this regard, and use a 3-step model, namely:
Formulation / provision of standards; Measuring / evaluating against it; Reaction to it.
This evaluation is done in a measurable format, indicating whether it is compliant, partially compliant or non-compliant. It is important that the department take responsibility and ownership for the activity.
4
A problem that the Nursing department experiences with the focus of the professional body, which is professional accreditation. But, what about quality? They want to consolidate Dr Vroeijenstijn's model with the existing accreditation exercise. The department of Nursing is used to and prepared for evaluation. They would like to critically investigate the model to see how it can be included and adapted according to the system they are used to. The department will provide Dr Vroeijenstijn with the necessary documents regarding their accreditation (Attached as Addendum 1). Dr Vroeijenstijn felt that the models are compatible. People will always experience the idea of formal measurement as bureaucratic. The department of Nursing feel that they are used to this form of evaluation, and do not experience it as being threatening. The instrument will be adapted specifically for the department and is not prescriptive in any way. No matter what instrument or model is used for SE, there are certain basic questions, which have to be included and reflected upon. The professional body have some standards, but academic standards are dynamic and must be adapted regularly to stay internationally competitive and comparable. The Quality Assurance Association (QAA) of England (www.qaa.ac.uk) developed benchmarking standards for 28 disciplines. It can be used as examples when we have to formulate our own standards. The South African Qualifications Authority (SAQA) is also in the process of formulating standards by means of the National Standards Bodies. But the central idea is that a faculty or department sets its own standard and measure themselves against it. We must strive not to have diverse and separate systems, but they should be consolidated to complement each other. The terminology needs to be cleared at the beginning of the exercise. The QCC can learn from Nursing in that they have come a long way in this regard. It is in the interest of RAU and its clients. The committee also expressed their appreciation to the department for their willingness to participate. The academic community needs to be kept informed and a culture of evaluation must be created. We do not have a choice, so the climate must be established and structures need to be formalised to manage the process. A committee must be appointed within the department of Nursing to manage and implement the process. The question of follow-up procedures was raised. Recommendations should be made for changes and improvements. It does not necessarily only mean an investment of money; some things can be changed without cost. It is not a punishing act, and departments should not feel threatened. The evaluation will also not be individual in nature, but a collective approach will be followed.
5
MEETING 4
Date: Tuesday, 27 February 2001 Time: 9:00 Venue: Conference room — Faculty of Economic and Business Management Group:
Prof D, provided the following introduction regarding the faculty: The Faculty of E have approximately 4000-5000 students (± 40% of the university's students). The faculty also indicated that they are predominantly output and result driven. They also have the lowest student-lecturer ratio. They are mainly concerned about the quality of the results, and concentrate less on the quality aspects of the process.
Dr Vroeijenstijn made the following remarks: How can I know my product is good, if I don't know that the production process is good? But he stressed that it is up to the faculty to decide how they want to ensure quality. Faculties often have questions and comments like: "Why should we do it?", "We don't have time", "It is bureaucratic nonsense", "It is a threat to my academic freedom", "Who do you think you are to come and tell me what to do?" etc. Most faculties, who expose themselves to such a SE, usually respond: "It was a hell of a job, but we are glad we did it!" Management should support the initiative, but they should not be seen as a control mechanism. South Africa should take a lead in this regard, seeing that government in the near future would enforce it. Faculties often blame low budgets or limited finances for low standards in quality, but it is not always a matter of money. Sometimes it is only attitudes that must be changed and thinking that needs renewal. But in the instances where money is needed, management should provide financial support as far as possible. Staff motivation is often problematic and a positive atmosphere should be created. During the implementation, staff should be motivated to rectify problem areas and to take corrective actions. Make it clear to staff members that it is also to improve their working conditions and levels of satisfaction. The aim of the exercise is not to close down any department, but to identify or analyse strengths and weaknesses. The focus is not on the negative, but to build and expand on the positive issues. An internal evaluation should be followed by an external assessment. The two would be linked and the external assessment should be seen as an instrument to check the internal quality. Departments should not do internal
6
evaluation merely for preparation for the external, but they should be aware of the departmental value of the SE. External advice and input are necessary to determine whether the department output meets the demands of the labour market. External forces should then influence, on a continuous basis, curriculum design and development. The department will play an active role in nominating the experts in the specific field. The programmes offered in a specific department are the starting point for assessment. All contributors to the program should be involved in the SE. Evaluation of research is also important, but we have to start somewhere, and quality in teaching and learning would filter through to research projects. The SE-exercise should be followed by a strategic planning session, based on the self-evaluation. This plan needs to be updated each year. The environment is changing so rapidly that a SE needs to be done at least every 5 years. The strategic plan will then be adapted accordingly. Management should rethink the role of the annual reports, and how the SE can be incorporated. The SE should be absorbed into it, and not be an additional burden and administrative exercise. It should be integrated in one report. Some departments within the faculty already have certain procedures in place. That should also be incorporated and used in the SE. Internal quality assurance systems within faculties should be put into place to determine, for example, student satisfaction. The evaluation of teaching by students should also be refined to be incorporated in the SE. The opinion of the labour market and Alumni should also be determined and used for strategic planning. Departments should not underestimate the importance of "small quality" e.g. being on time, preparation for lectures etc. Regarding the matter of undergraduate students being incompetent to evaluate lecturers, they are the 'clients, and their perceptions and experiences influence their studies and decisions. The SE-manual is not a questionnaire with certain questions that needs to be answered. It is more of a qualitative approach of reporting on a certain issue prompted by means of formulated questions.
MEETING 5
Date: Tuesday, 27 February 2001 Time: 12:00 Venue: Conference room — Faculty of Science Group: F
7
SE should not be seen as a bureaucratic system that is out of the control of the faculty or department. It is important and valuable for internal development. It is time-consuming and demands a lot of effort, but eventually it will be made compulsory from government side. We need to prepare ourselves internally for this. Faculties and departments must establish an internal quality assurance structure. Each department can adapt the instrument provided to suit their specific needs and situation. It will not be enforced and must be seen as a guideline. The exercise should be repeated every five years. It should also be followed up by the formulation of a strategic plan. Strengths and weaknesses must be identified. The weaknesses would be prioritised to determine which are the most important. The instrument should not be seen as questionnaire, but an aid to guide a department to reflect qualitatively on certain activities and issues within the department. The faculty already have some systems in place and the SE should be incorporated with the existing practices. The support and commitment from management is stressed once again. The role of the QCC should also be explicit and their activities visible on campus. The departments, which are taking part, should appoint an internal work group. They must have access to the necessary information. A draft report should be presented to the whole department for information and comments. Their experiences should be shared with other departments. The recommendations made must be followed up. It does not necessarily cost money to do this. Sometimes a change in attitude is required, but management must be committed to provide funding where possible. The tension between decentralisation vs. centralisation was discussed. A balance between the two is needed, but the responsibility lies within the department to establish their own criteria for SE, so that the staff can accept ownership for the process. Other institutions in South Africa are already actively involved in SE, e.g. Stellenbosch and Pretoria. We are not in the position, as yet, that government forces it upon us. The Netherlands would never have engaged in SE, was it not forced on them from the outside. Institutions in South Africa should take the initiative and prepare themselves for the inevitable. The HEQC promised in the Founding document that they will have a "light touch" if institutions are actively involved in audit activities.
MEETING 6
Date: Tuesday, 27 February 2001 Time: 14:00 Venue: Conference room — CHES
8
mug: Task Team 7 & QCC secretariat
Regarding the document of Task Team 7, Dr Vroeijenstijn had the following comments:
The context should be explained from an international, national and RAU perspective. The development of policy documents is a priority. Examples can be found on the website of the QAA (Quality Assurance Agency) of the United Kingdom. It is also important that it should be implemented on faculty level and that the QCC is not seen as a centralised bureaucracy.
MEETING 7
Date: Wednesday, 28 February 2001 Time: 9:00 Venue: Boardroom, A ring 3 Group: FOTIM
The notes of the workshop as provided by FOTIM are attached as Addendum 2. The slide presentation is attached as Addendum 3.
The following additional comments were made regarding SE:
SE should be done holistically, not in a fragmented manner. Certain problems and questions are experienced when engaging in a SE-exercise, e.g.:
Why should we do it? We never had complaints before. We don't have time. SE is seen as a firing or closing down tool. It is seen as a threat. The involved parties do not know why they are doing it. The main aim should be internal quality improvement. It is not a punishment. Dishonesty. What would the consequences be? Weaknesses should be discussed thoroughly. We need money to improve. Management does not support us. All aspects need to be evaluated, including management structures. It is seen as individual appraisals. Unexpected outcomes. Lack of data, results in 'white spots' where answers are not available. Provision should be made for the development of an instrument to address the 'white spots'.
9
There should not be two systems of SE in one country for internal and external assessment. The following are important perspectives captured in the January Founding document of the HEQC:
The importance of the relationship with professional bodies regarding accreditation procedures. The idea of Ileamerships' to build capacity. Even without external assessment, SE is necessary for personal development and planning. External assessment is often used as a political instrument. It is meant for the assessment of quality and should be independent of policies. Can we convince the external bodies that this is the correct instrument to use so that there is a correlation between internal and external assessment procedures? Public and private providers should all be subjected to accreditation.
The main aim for internal and external assessment is improvement. The process and the product need to be assessed. It you take care of the process, the product will be successful. But the desired product needs to be determined beforehand. The problem in higher education (HE) is that we need to define clearly what kind of graduate or output we expect, which is not an easy task. We are too process-orientated. The output should be clearly defined and then we must determine what kind of skills, knowledge and attitudes we need to achieve that specific output. The labour market is only interested in the product. Quality SE is group motivated with clear explanations about what is expected. Certain instruments should be in place to collect and access the required data. The SE report is addressed to and discussed within the department. It is also important to direct it to the faculty or body on central level. Recommendations should be addressed. The role of the Quality Unit (QU) is to monitor the process, motivate staff and facilitate the process. They should provide assistance regarding the process to follow. They should organise training sessions, but have no say in the assessment itself. The department set their own criteria. The QU should also assess and analyse its own situation. It is definitely problematic to continue with the process before clear guidelines are established by the HEQC, but on the other hand, institutions can't sit around and wait for governmental intervention. Quality depends on the setting of criteria; the department or institution decides what is acceptable for them. There can also be conflict between stakeholders, but they must strive to reach a balance between the different requirements and points of view.
10
The support needed from central management is stressed once again. They should make it possible and motivate the staff to participate. Quality is made at grass root level, and the departments are still responsible for their own initiative, but central management should provide the necessary instruments. They should be involved in the follow-up procedures, providing support and resources where possible. The idea of a 'contract' between the department and management was introduced, as to who plays which role in the follow-up procedure. The setting of criteria should be done within a certain framework. Benchmarking should also play a primary role in SE. Departmental workshops should be planned by the departments themselves and not prescribed. The QU should assist in training and support. Why do we use models when venturing in SE?
It helps us to direct our thinking, but leaves the opportunity to formulate and determine our own goals and aims. It helps us to discover 'white spots'. It helps the work group to structure the exercise.
MEETING 8
Date: Thursday, 1 March 2001 Mpg: 8:00 Venue: Faculty of G Group: Faculty G
The SE must not be seen as a bureaucratic process, but as something for personal benefit and improvement. Higher education becomes more and more competitive. SE is a cyclical process and the instrument can be adapted for the specific department or faculty. It must be seen as a guideline, and not a prescriptive questionnaire.
A professional body is accrediting the Faculty of G comprehensively. This will take place in August this year. The outcomes-based format will specifically be investigated this year. A member from the academic sector and industry constitutes the external panel. Dr Vroeijenstijn commented that producing material to an external body is not SE. There is no opportunity to reflect and make recommendations for future actions. The professional body also only look at undergraduate courses. Dr Vroeijenstijn will have a discussion with the Faculty of G in May to precede and assist with the accreditation, taking place in August.
1 1
MEETING 9
Date:
Thursday, 1 March 2001 ra_gn :
10:00 Venue:
Boardroom — H Group:
H
SE is done for personal benefit, not only because of external pressure. It is not a bureaucratic process, and does not inhibit or threaten academic freedom. The faculty / department must determine where they fit in within the broader university strategic plan. It is sometimes a painful process, but must be seen against the background of competing institutions and the importance of internationalisation. There is a definite need in South Africa to raise the levels of quality. A paradigm shift is needed and a change in mentality with regard to the idea of an individual academic moving towards the notion of a business-like operation. Higher education will not survive without making this conceptual shift. This needs to be attended to in a structured manner. SE is a formalised quality 'test', and we need to do it on our own initiative and not wait around for the Ministry for force it upon institutions.
A new I-programme was instituted in 1997. The success of the course needs to be determined. The faculty feel that the manual can assist them in this evaluation. The faculty also have a journal that is losing subscriptions, and they want to determine why. This is definitely a quality issue which needs to be addressed, also in terms of subsidy. The faculty wants to initiate an internal assessment, and the manual can be adapted for their specific needs. The faculty must have ownership and the process must be intemalised. The role of the annual report needs to be re-evaluated in this regard.
MEETING 10
Date:
Thursday, 1 March 2001 Time:
12:45 Venue:
Boardroom — Faculty of 3 Group: Faculty of 3
Higher education institutions become more and more competitive in terms of national and international developments.
12
SE should be introduced as a continuous process and instrument to assess the quality of programmes and departments. The faculty should be participating in adapting the manual as a thinking instrument about quality. It is actually nothing new, but just a more structured way of reflection. It is also not a questionnaire that needs to be completed. The focus needs to shift from the idea of the individual professor, to a completely joint enterprise including all stakeholders. There are certain problems and concerns involved, e.g. the reasons why it should be done, resistance, lack of time, overloaded personnel, bureaucracy, support from management, workload etc. A corporate identity should be promoted and programmes must also be viewed as joint enterprises. The definition we accept for quality needs to correspond with that of the national bodies. Benchmarking should be used in this regard The faculty of Arts consists of a variety of diverse programmes and departments. The SE would be done by means of investigating programmes, not departments. The focus is not only on determining weaknesses, but to build on and promote strengths.
MEETING 11
Date: Thursday, 1 March 2001 Time: 15:00 Venue: C les 303 Group: Non-academic department heads (Support units)
Non-academic departments are diverse and unique in nature and a manual specifically for that has not been developed. It would have to be tailor made for each unit. The support units (SU) have a large impact and role to play in the overall quality of an institution. It will be forced on the institution eventually by government, but it is also valuable and beneficial for self-improvement. The nature of higher education is becoming more and more competitive, and we see more and more international institutions entering the South African context. With regard to SU, it is very important to determine the satisfaction of the clients, and whether goals are achieved. The manual for SU will mainly focus on five areas to evaluate:
Goals and objectives; Resources used to achieve these goals; The procedures followed; The task performance; and
13
Feedback from and satisfaction of clients. Questionnaires must be compiled to determine the satisfaction of the clients. This can be done in a combined effort among different SUs. It is basically doing a SWOT-analysis. Sometimes an objective facilitator is needed to manage the process, or the unit itself can do it. The following SUs volunteered for the pilot SE project:
The library, Public relations and Sport bureau.
A holistic approach must be followed. The academic departments and SU influence each other and have a major impact on the overall quality of the institution.
MEETING 12
Date: Friday, 2 March 2001 lime: 8:00 for 8:30 Venue: A les 103 fi: RAU workshop
The slide presentation from the workshop is attached as Addendum 4.
MEETING 13
Date: Friday, 2 March 2001 Time: 14:00 Venue: C les 303 Group: CHES
Dr Vroeijenstijn will train the first two external teams. The CHES is responsible for the training of the other external committees and departments. The manual for SU will mainly focus on five areas to evaluate:
Goals and objectives; Resources used to achieve these goals; The procedures followed; The task performance; and Feedback from and satisfaction of clients.
An internal work group needs to be established. The staff of the university is our clients. We serve students via lecturers. The centre must reflect on its value for the RAU. What will they miss if the centre is no longer in existence?
14
MEETING 14
Date: Friday, 2 March 2001 Time: 15:00 Venue: Tearoom — Department K Group: Department K
The staff had to adapt to the merging of L departments. They felt that it was meaningful to combine the two departments because there were overlaps between programmes. They also feel they have not reaped any rewards or follow-up regarding the initial SE exercise of the university. Dr Vroeijenstijn pointed out that it was not really a SE, but merely a questionnaire that had been completed. An internal committee must be established to manage the process. They must have access to information and will be trained when Dr Vroeijenstijn visits us in May. It is preferable to have a student on the committee. The department enquired whether it is possible to obtain an example of a SE report done in a similar department Dr Vroeijenstijn will investigate this matter.
MEETING 15 - Debriefing
Date: Saturday, 3 March 2001 "Inc 15:00 Venue: 133's Midrand
vug: CHES
A meeting must be organised with the 7 volunteer departments. They need to appoint their internal work groups and convenors. We must investigate the manual and send suggestions to Dr Ton. He will complete the manual for SU.
15