key issues in medical education: implications of educational technology trends

10
Indian J Pediatr 1993; 60 : 729-738 Key Issues in Medical Education : Implications of Educational Technology Trends AlistairStewart Technology Transfer Centre, Dundee hzstitute of Technology, Dundee I n an overview of themes in medical edu- cation, Henry Walton ~ presented what he considered to be the key issues in medical education. They included : 1. The medical school as an institution, in which he singled out the resistance to change as a prominent attribute; 2. Science or art, in which he high- lighted a diminished concern with patients as people as an undesired accompaniment of progress in science and its adoption in clinical medicine; 3. Entering medicine, which he sug- gested as being the most pressing concern in medical education, world wide, arguing for intensive research on methods of selec- tion; 4. The curriculum, the construction of which be recognised as being at the centre of what medical schools are about, and the areas of interest he identified as concern with objectives, integration, incorporating new subjects; training settings, and problem-based learning; 5. Motivation, in which he recognised the problem of students being demotivated by their studies; 6. Medical teaching, in which he ac- knowledged that, since medical teachers Reprint requests: Dr. Alistair Stewart, Director, Technology Transfer Centre, Dundee Institute of Technology, Bell Street, Dundee, DDI I HG U.K. are largely amateurs, an interest in teach- ing must be encouraged; 7. Traditional/new medical schools, in which he pointed out that, since the vast majority of the world's medical schools are 'traditional', any development and reform must occur in these schools, and that 'new' medical schools are usually characterised by community-oriented programmes and the use of a problem-solving educational process; 8. How doctors diagnose, in which he identified inconsistency between the way students are taught to diagnose and the way experienced physicians actually do so; and 9. Other issues, among which he included primary care, evaluation and assess- merit, postgraduate education, and continuing education. The paper covers a wide range of topics of concern, but does not attempt to analyse in order to identify critical elements and to set these within a conceptual framework. The purpose of this present paper is to outline current trends in educational technology, and to establish a conceptual framework within which the key issues in medical education can be addressed. The educational technology trends which will be examined are those concerned with cognitive and conceptual advances and those concerned with technology-based developments.

Upload: alistair-stewart

Post on 16-Aug-2016

219 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Key issues in medical education: Implications of educational technology trends

Indian J Pediatr 1993; 60 : 729-738

Key Issues in Medical Education : Implications of Educational Technology Trends

AlistairStewart

Technology Transfer Centre, Dundee hzstitute of Technology, Dundee

I n an overview of themes in medical edu- cation, Henry Walton ~ presented what he considered to be the key issues in medical education. They included :

1. The medical school as an institution, in which he singled out the resistance to change as a prominent attribute;

2. Science or art, in which he high- lighted a diminished concern with patients as people as an undesired accompaniment of progress in science and its adoption in clinical medicine;

3. Entering medicine, which he sug- gested as being the most pressing concern in medical education, world wide, arguing for intensive research on methods of selec- tion;

4. The curriculum, the construction of which be recognised as being at the centre of what medical schools are about, and the areas of interest he identified as concern with objectives, integration, incorporating new subjects; training settings, and problem-based learning;

5. Motivation, in which he recognised the problem of students being demotivated by their studies;

6. Medical teaching, in which he ac- knowledged that, since medical teachers

Reprint requests: Dr. Alistair Stewart, Director, Technology Transfer Centre, Dundee Institute of Technology, Bell Street, Dundee, DDI I HG U.K.

are largely amateurs, an interest in teach- ing must be encouraged;

7. Traditional/new medical schools, in which he pointed out that, since the vast majority of the world's medical schools are 'traditional', any development and reform must occur in these schools, and that 'new' medical schools are usually characterised by community-oriented programmes and the use of a problem-solving educational process;

8. How doctors diagnose, in which he identified inconsistency between the way students are taught to diagnose and the way experienced physicians actually do so; and

9. Other issues, among which he included primary care, evaluation and assess- merit, postgraduate education, and continuing education. The paper covers a wide range of topics of concern, but does not attempt to analyse in order to identify critical elements and to set these within a conceptual framework. The purpose of this present paper is to outline current trends in educational technology, and to establish a conceptual framework within which the key issues in medical education can be addressed.

The educational technology trends which will be examined are those concerned with cognitive and conceptual advances and those concerned with technology-based developments.

Page 2: Key issues in medical education: Implications of educational technology trends

730 THE INDIAN JOURNAL OF PEDIATI~dCS

Analysis of Key Issues in Medical Education

A number of the issues which Dr. Walton raises could be considered as being related to the aims and objectives of courses in medical education or the goals of the institution. For example, the issue of diminished concern with patients as people is linked to both institutional goals and specific aims and objectives of the course. Similarly, the distinction between new and traditional medical schools is a distinction based on institutional mission and goals; it could, of course, also involve other factors such as readiness with which an institution is willing to change. Resistance to change, another issue revised by Dr. Walton, is primarily an issue of institutional mission and goals.

Assessment of students is mentioned as a specific concern, but assessment could be extended to take into consideration another of the issues, viz. the selection of students for medical school.

Several of the issues identified fall readily within the scope of curriculum design. In addition to the explicitly stated issues, those concerned with diagnosing & problem-solving, primary health care and community oriented programmes, and motivation, can legitimately be subsumed under curriculum issues.

The only other issue which Dr. Walton raises is that of teaching methods. Thus, all the issues could be considered as falling within tl~e areas of institutional goals/ mission, aims & objectives, student assessment, curriculum design, and teaching methods. It is in relation to these areas that implications from educational technology developments will be explored.

1993; Vol. 60. No. 6

Trends in Educational Technology

Before outlining recent developments in educational technology it is probably worth restating the basis conceptual framework within which educational tech- nology operates.

Cox and Ewan 2 mention the systems approach to the development of teaching and claim to have based their textbook on it. They argue that :

"Systematic design is a popular approach to medical education, and is a useful framework to bring to medical education" (p vi)

However, although they concern them- selves with components of the systems approach as it affects the teaching and learning process, they do not adopt the systems approach in its totality or high- light its systemic nature.

The Systems Approach

The systems approach has characterised medical education around the. world for the past 15 years. There has been an emphasis on the identification of objectives; there has been an on-going effort to improve teaching; and there has been a continuing interest in the development of assessment instruments.

One way of representing the systems approach is shown below.

Page 3: Key issues in medical education: Implications of educational technology trends

!993; Vol. 60. No. 6 THE LNDIAN JOURNAL OF PEDIATRICS 731

It is worth considering each of these phases in some detail.

Analysis

Surely the key issue which needs to be ad- dressed in medical education is identifica- tion of what a practising physician needs to be able to do. As noted in one of WHO's public health papers (No 68), 3

"Defining professional competence is the cornerstone upon which . . . a programme of medical education is built. Unless this task is approached both thoughtfully and systematically the medical curriculum is more likely to be a reflection of faculty interest than of student and public needs" (p21)

There are various ways in which the pro- fessional competence of the physician can be identified including analysis of what a physician does, either by self-reports, by observation, by task analysis or by critical incident techniques. But, as also noted in the WHO public health paper (No 68) :

"In the end it is the health care needs of the community, and the resources available to meet those needs, that should provide the principal ditec- tional signals in building a curricu- lum".

The preparation of physicians is an expen- sive exercise; governments have a right to expect their investment in health man- power development to lead to appropriate provision of health care through relevantly qualified personnel, and institutions which train physicians and other health person- nel must accept accountability to their pay- masters by developing curricula which re-

cognise the kind of health care needs in society, and which equip practitioners to be able to deliver such care.

But what kind of skills or competences is fllis analysis phase likely to identify? Without going into detail on the myriad of skills and competences which will, un- doubtedly, arise, it should readily be ac- knowledged that there will be abilities as- sociated with what are usually described as knowledge, attitudes and skills. Most tasks which the physician will undertake will probably involve abilities in all three areas, but for convenience it is usual to consider each area separately, and perhaps most attention is usually given to the area of knowledge.

The important aspect about the knowl- edge area is that individuals can demon- strate different levels of ability, from the ability merely to recall information to the ability to analyse problems and use not only information, but apply an under- standing of concepts and principles, in the solving of problems. It could be argued that any practising physician needs to be a medical problem solver (and that would probably show up in the identification of competences), so medical education needs to be concerned with bringing under- graduates to this level of problem solving.

But the evidence from medical schools suggests that much of the curriculum and its assessment is concerned only with the recall of information; lectures provide an opportunity to present factual information, and MCQs presented in examinations usu- ally measure only recall of facts. Improve- ment in teaching and experimentation with multiple choice assessment have both been successful, but neither is tackling the level of performance identified. Identifying the competences associated with the practice

Page 4: Key issues in medical education: Implications of educational technology trends

732 THE INDIAN JOURNAL OF PEDIATRICS 1993; Vol. 60. No. 6

of medicine is important, therefore, be- cause it influences the other two phases of the systems approach, in the design of learning experiences and in the assessment of student performance.

Design

Identification of the competences required by the physician leads to the identification of specific learning outcomes or objectives which must be achieved by the student. It will also lead to identification of the inter- relationships between objectives; which objectives need to be achieved before mov- ing on to other objectives, which concepts are pre-requisite to particular principles and in what sequence topics need to be dealt with. In other words, the criteria for the design of the learning experiences are beginning to emerge, the curriculum is be- ginnhlg to be developed. The curriculum is not concerned with content, it is concerned with competence.

Competence then determines not only content, but the nature of content; it determines not only the curriculum but the curricular experiences which need to be offered to the student.

But all this raises issues regarding how students learn. For years, medical schools have been helping teachers to be better teachers, but is it possible to have a theory of teaching without, first, having an ade- quate theory of learning? Might it also be necessary to help students to be able to learn - learning how to learn - particularly since education in the school system was probably concerned with remembering factual information, and education in the medical schools is much more concemed with higher level cognitive abilities such as problem-solving, and this will be dis- cussed later in this article.

Implementation & Evaluation

In this phase the design is put into practice and the outcome is evaluated. Evaluation includes assessment of student perform- ance against identified objectives and com- petences. If the competence or objective calls for ability at the level of problem- solving, then the assessment must measure at that level. It is unlikely that MCQs are readily available for testing at that level,* and the student will have to be exposed to the problem and his ability in solving the problem assessed. The use of OSCEs, s has been highly successful in measuring stu- dent ability to solve certain types of prob- lem, as has the use of simulations. 6

Evaluation is not, however, limited to student assessment; it includes evaluation of all aspects of the system and of the sys- tem as a whole, and can be concerned in an on-going, continuous way (formative evaluation) as well as in a final (summa- five) way. If, for example, students experi- ence difficulty in achieving a particular performance, consideration needs to be given to the design of the test instrument - did it measure reliably what it set out to measure; or to the learning experiences - were they designed carefully enough to facilitate achievement of that particular ability; or to the identification of the objec- five - was it derived from valid analysis and were the criteria of achievement realis- tic; or to the entry ability of the student - did he/she have the pre-requisite knowl- edge, attitudes, or skills. And the impor- tant feature of all this is that they are inter- related. It is not possible to change some- firing in one part of the system without also affecting other parts of the system.

This, then, is the systems approach to teaching and leaming and the design of

Page 5: Key issues in medical education: Implications of educational technology trends

1993; Vol. 60. No. 6

instruction; it offers a particular perspec- tive for the analysis of key issues in medi- cal education, and it offers a framework for consideration of the inter-relationship of these issues.

Recent Developments

Acquisition of knowledge. Perhaps the most important aspect of the develop- ments in cognitive psychology has been the identification of the structured and or- ganised nature of knowledge in memory. It is generally concluded that :

"Knowledge is structured by individuals in meaningful ways which grow and change over time".

"The way in which knowledge is stored is related to the way knowledge is encodedwhen it is earned".7

From this it is clear that learning is an individual exercise. There is no way in which the teacher can "impart knowledge'. He/she may impart information, but knowledge will be built-up by the learner depending on what is already known (and in what form) and on the circumstances surrounding the particular learning exercise. The responsibility for the teacher, therefore, is to help the learner to learn : ~aching must facilitate learning.

Student learning in higher education. While teachers in higher education are of- ten of the view that they are teaching their students to be problem solvers, and com- petence-based learning may be widely dis- cussed as an objective of medical educa- tion, the evidence available concerning the outcome of higher education is quite dis- turbing.

THE INDIAN JOURNAL OF PEDIATRICS 733

in a report on the largest programme of research into student learning ever carried out in the United Kingdom, the authors concluded that :

"The insistent contrast between students' ways of s tudying . . , is between deep and surface or meaning and reproducing approaches to learning ''8

This distinction between deep and surface approaches to learning is a reference to earlier work done in Sweden and, since, in many parts of the world, in which it is re- cognised that :

"'In the case of surface-level processing, the student directs his attention towards learning the words - he has a 'reproductive' conception of

learning resulting in a rote- learning strategy.'"

"In the cases of deep-level processing, the student is directed towards the intentional content of learning material - he is concerned with comprehending what the author wants to say. ''9

There is no shortage of information on the tendency of students to be concerned with the memorisation of material rather than with a deeper understanding of its meaning, and this appears to be the case more often with science students than with arts students.

"'Science students tend to be motivated by vocational concerns and to adopt surface-level reproductive study methods."

"Arts students show intrinsic interest in their courses and adopt a deep- level approach to their work. "'1~

Page 6: Key issues in medical education: Implications of educational technology trends

734 THE INDIAN JOURNAL OF PEDIATRICS 1993; Vol. 60. No. 6

If this is the case, and there certainly seems to be evidence for it, it may well be that the nature of science-based courses has some- tt-dng to do with it.

"Students' perceptions of teaching methods and assessment can determine whether deep or surface approaches are adopted. ' 'n

Are medical teachers concerned with the imparting of factual information? If so, the student quickly realises that not too much thinking is required. All he has to do is try to remember the facts. At examination time this view is reinforced when he finds that the questions are concerned with re- gurgitation of memorised information. On the other hand, if the teaching/learning sessions are designed to promote thinking ability and the examinations are about problem solving, the student realises that a deeper approach has to be adopted. Fortu- nately, there is evidence that students' ap- proaches to learning can be modified.

"Depth of processing can be influenced by task orientation. ''12

The responsibility lies with the teacher to ensure that the student has the opportu- nity to think for himself, whether in the classrooms, clinical area, or examination Setting.

"Students relying on reproducing information allow staff to define learning tasks. Students looking for meaning inter-act critically with what they are learning. ''8

Of course, the aim to teaching in higher education is to try to bring about higher- level cognitive abilities. A report on uni- versity teaching in U.K. highlighted this.

"The aim and nature of the under- graduate course should not be only or primarily to equip the student with knowledge, but also, and more impbrtanfly, to teach him to think for himself and to work on his own. 14

Nevertheless, it has to be acknowledged that many students never seem to be able to think for themselves or to work on their own. They pass examinations and gain their degrees by remembering information. But an emphasis on meaning and critical thinking is essential not only for under- graduate courses; it lies at the heart of the whole notion of life-long learning and must surely be a pre-requisite for the indi- vidual who is likely to be involved with continuing professional education. What do our courses do to encourage thinking abilities?

"Relatively little attention has been given to the teaching of thinking skills...to the teaching of the skills involved in such higher -- order activities as reasoning, creation thinking, and problem solving. '''5

Why not? Certainly in medical education the successful practitioner would be ex- pected to be able to demonstrate these abilities. Of course, there needs to be knowledge, but there also needs to be thinking ability.

"Thinking is essential to the acquisi- tion of knowledge. Knowledge is es- sential to thinking." "Thinking ability and knowledge are the warp and woof of intellectual competence, and the development of either to the neglect of the other will produce a less than superb fabric. ''15

Page 7: Key issues in medical education: Implications of educational technology trends

1993; Vol. 60. No. 6

A particular thinking ability associated with medical education must surely be what has been called scientific thinking which uti]ises inductive and deductive rea- soning in the application of concepts and principles to the solving of identified prob- lems. 16

There is a responsibility, therefore, on the teacher, to be able to help his students not only to acquire knowledge but to be able to use that knowledge in the solving of problems.

Technology-based Learning

Recent developments in technology-based earning have almost always involved a microcomputer. Linked to the microcom- puter have been such audio & visual me- dia devices as videodisc or CD ROM, ena- bling rapid and random access to sound and pictures in both analogue and digital forms. The essential characteristic of these technology-based learning systems is that they have made possible a considerable degree of interactivity between the learner and the learning materials.

As Barker & Tucker have pointed out, 7 interactive learning, defined as "learner- centred learning using a multimedia ap- proach", is really a process rather than a technology.

The digital revolution enabling sound and vision to be digitised, for example as in digital video interactive (DVI), has made possible the delivery of interactive multimedia without the use of analogue sound and" vision sources and utilizing only a microcomputer (although the digital information needs to be on a CR-ROM be- Cause of the vast amount of digital infor- mation involved). This DVI technology al- lows all types of information, including

THE INDIAN JOURNAL OF PEDIATRICS 735

full-motion video, still images, audio, text, and graphics, to be stored and distributed on an all-digital source. The microcom- puter is tumed into a multimedia tool and the interactivity capability of the computer can be applied to all types of media.

"Applications developers will take this capability and build new, creative ways to educate ''18

As already noted, however, it is not the technology which is important; it is the process, and particularly theprocess of in- teraction between the learner and the ma- terial.

As Laurillard notes 19 :

"Interactive media . . . . . require the teacher to think in a learner-centred way. The point of these new instructional forms is to make the learners active rather than to be a passive recipient of knowledge, and it follows that the activities they undertake mus be "mathemagenic! - they must "give birth to learning'

The possibilities for technology-based learning depend for their successful exploitation on their capability to "give birth to learning" and that, in turn) depends on the developers of materials J~or such systems having an understanding of how people learn as suggested by research in cognitive science, and on their having an understanding of the design of learning materials and learning experiences as suggested by the systems approach to teaching and learning.

Implications for Medical Education

Developments in educational technology,, whether cognitive and conceptual or ted1-

Page 8: Key issues in medical education: Implications of educational technology trends

736 THE IN DIAN JOURNAL OF PEDIATRICS 1993; Vol. 60. No. 6

nological, have relevance for medical edu- cation since they highlight the importance of instructional design; development and delivery which is geared towards the fa- cilitating of learning and enhancing cogni- tive ability.

Obviously the curriculum is an issue which is, and which will continue to be, central to any consideration of medical education. Student assessment is also obvi- ously central to a discussion of issues. The medical teacher needs to be able to create learning experiences which will facilitate the construction of meaning, and needs to develop assessment procedures which measure the organisation and utilisation of knowledge. As has been observed,

"The curriculum and its assessment are the key factors affecting the quality of student learning". 2~

There is a relationship, therefore, between the curriculum, student assessment, and student learning.

But the quality of the learning experi- ence and the quality of the learning out- come require a concern not only with the methods of teaching and assessment, but with the ways in which the student uses his/her cognitive abilities. There is need for a special emphasis on thinking skills; they need to be emphasised in every disci- pline area along with an emphasis on deep processing. Student learning and thinking are key areas in medical education, but it should be noted that research suggests that,

"'It is very much harder to teach people how to learn than it is to teach them facts". 21 "'Stressing the development of thinking can only come about at the cost of reducing the amount of traditional content covered". 22

This raises another issue identified earlier, namely, how to cope with new material. The problem with the present curriculum (any curriculum!) is that it is concerned with content; inevitably content expands either within a discipline or through the addition of new disciplines. Ultimaely, there has to be selection of content. But selection should not be on the basis of how well heads of department can argue in commit- tee; it should be on the basis of essential concepts and principles to enable the stu- dent to think about problems and bring his/her understanding of concepts and principles to bear on the problem. Since the student knows how to learn, he/she can readily find out about content not included in the course.

The issue of integration is not a question of should it be done or should it not be done. The purpose of integration should be to provide relevance for the student and to enable the student to draw together infor- mation from several content areas in rela- tion to a larger topic since, in due course, that is the way the medical problem will be tackled. It is related, therefore, to the kind of objectives identified in the analysis phase and dealt with in the design phase. It does not exist in isolation.

Selection criteria, which is seen as a very pressing concern around the world does need consideration within the framework suggested. In most schools of medicine, se- lection is based primarily on performance in school tea_vin~ examinations or in an en- trance examination, neither of which is normally measuring any more than the student 's ability to recall information. Modification of these tests to measure thinking skills and problem-solving skills would be a helpful first step, since it would at least match student ability with the kind

Page 9: Key issues in medical education: Implications of educational technology trends

1993; Vol. 60. No. 6 THE INDIAN JOURNAL OF PEDIATRICS 737

of learning experiences to which they ~vould be exposed in medical school. But ~aere must also be an emphasis on atti- tudes and aptitudes since the practice of medicine is so much more than a cognitive ability. Culture or social constraints may pose problems for selection based on these criteria, particularly where judgement is ~een to be subjective, but objective meas- ures do exist and can be utilised.

The issue of training setting is difficult to deal with, particularly when agencies such as WHO are pressing for community- based training. From the WHO perspec- tive, where there is concern with primary health care and its contribution to "Health t0r All by the year 2000", and where the maldistribution of health care provision between rural and urban areas and be- tween community and hospital settings is acutely obvious, it is understandable that, ~vith an emphasis on preventive and reha- bilitative health care rather than on cura- tive care, the setting for training ought, at least, to include the community. But again, tt depends upon the competences and ob- lectives identified in the analysis phase. If the objectives call for competence in the :ommunity, training in the community is ~nevitable; if the objectives call for other :0mpetences associated with curative care ~nd advanced medical science, training ,rill have to be in an appropriate setting.

CONCLUSION

(ey issues in medical education do not, as soften apparently believed, exist in isola- E0n, and cannot be dealt with in isolation. 7be issues are only symptoms; they are ldlcative of deeper relationships which '~st within a system of teaching and ~Orning. Rather than deal with the syrup- Jms, it is better to understand the nature

of the system and what is giving rise to the symptoms. The systems approach to teach- ing and learning not only gives an appro- priate framework for analysing need, de- signing solutions and evaluating outcome, but of putting together within a manage- able structure the various pieces of what might previously have appeared to be an educational jigsaw.

Ideas about the nature of learning and the development of cognitive skills, coupled with the development of interac- tive methods of delivery make it possible for medical educators to help their stu- dents achieve the high level medical prob- lem-solving abilities identified in analysis of required competences.

REFERENCES

1. Walton H. Overview of themes in medical education. In : Goodland S. Education for the Professions, Society for Research into Higher Education, 1984.

2. Charvat J, Mc Guire C, Parsons V. A Review of the Nature and Uses of Examina- tions in Medical Education, Public Health Paper No. 36, Geneva, World Health Or- ganisation, 1968.

3. W.H.O. Competency-Based Curriculum De- velopment in Medical Education. Public Health Paper No. 68, World Health Or- ganisafion, 1978.

4. McGuire C. A process approach to the construction and analysis of medical ex- aminations, Journal of Medical Education 1963; 38 (7) : 556-563.

5. Harden RM, Gleeson FA. Assessment of clinical competence using an objective structured clinical examination (OSCE). Medical Education, 1979; 13 : 41-54.

6. Barrows HS. Multiple stations. In : Hart IR, Harden RM, eds. Further Developments in Assessing Clinical Competence, Montreal, Heal Publications, 1987.

Page 10: Key issues in medical education: Implications of educational technology trends

738 THE INDIAN JOURNAL OF PEDIATRICS 1993; Vol. 60. No, ~

7. Romberg TA, Carpenter TP. Research on teaching; and learning mathematics, In : Wittrock MC, ed. Handbook of Research on Teaching, 3rd edition, MacMillan Publish- ing Co., 1986 : 850-873.

8. Entwistle NJ, Ramsden P. Understanding Student Learning, London, Croom Helm, 1983.

9. Marton F, Saljo R. On qualitative differ- ences in learning. British Journal of Educa- tional Psychology 1976; 46 : pp 4-11 & 115- 127.

10. Watkins & Mattie 1981. 11. Ramsden 1992. 12. Ballstaedt SP, Mandl H. Influencing Depth

of Processing in Reading. Paper presented at the Annual Meeting of the American Educational Researcb Association, Chi- cago, April 1985.

13. Biggs JB, Rihn BA. The effects of interven- tion on deep and surface approaches to learning. In : Kirby JR, ed. Cognitive Strategies and Educational Performance, London : Academic Press, 1984.

14. Hale Report. UniversTty Teaching Methods, London, HMSO, 1964.

] 5. Nickerson RS, Perkins DN, Smith EE. The

Teaching of Thinking, Lawrence Erlbaum Associates, 1985.

16. Gunn MCA. The Development of Scientific Thinking in Egyptian Children Through Non-formal Educational Settings Utilising Teacher-Mediated Resource--Based Learning Materials. Unpublished Med dissertation, University of Dundee, 1984.

17. Barker J, Tucker RN, eds. The Interactive Learning Revolution. London : Kogan Page, 1990.

18. Bunzell MJ, Morris SK. Multimedia Appli- cations Development, New York : McGraw- Hill, 1992.

19. Wilson JD. Student Learning in Higher Education, London, Croom Helm, 1981. "

20. Cox KR, Ewan CE. The Medical Teacher, Churchill Livingstone, 1982.

21. Hallet DH. Understanding students who don' t understand maths. In : Akst G, ed Improving Mathematics Skills, Jossey-- Bass Inc. 1981 : 3-7.

22. Lochhead J. Problem solving for rote learners. In : Akst G, ed. Improving Mathematics Skills, Jossey-Bass Inc., 1981 : 12-26.