reflection on the development of a reflective assessment

7
Medical Teacher, Vol. 19, No. 2, 1997 Reflection on the development of a reflective assessment JENNYROUTLEDGE, MOYA WILLSON, MARGARET McARTHUR, BARBARA RICHARDSON & RICHARD STEPHENSON The School of Occupational Therapy and Physiotherapy, University of East Anglia, Nomich, UK SUMMARY This paper highlights how the School of Occu- pational Therapy and Physiotherapy at the Universig of East Anglia (UEA) has identified reflection as a major component of professional development within the undergraduate curriculum. It describes the evolution, development and implementation of a portfolio-based, reflective method of assessment. Discussion of this experiencefrom both staff and student perspectives identifies several interesting issues. Conrideration of this process points to a professional development of both members of faculty students. and In the vaned topography of professional practice, there is a high, hard, ground where practitioners can make effective use of research based theory and techniques, and there is a swampy lowland where situations are confusing ‘messes’ incapable of technical solution. (Schon, 1991, p. 42) Introduction Education for the professions must prepare students for the changing world in which they will practice. Assessment should reflect the students’ qualities and achievements which are actively rewarded by the system (Rowntree, 1977). Assessments, therefore, should focus on those abili- ties which are important in the field of practice. The School of Occupational Therapy and Physiother- apy at the University of East Anglia (UEA) has identified reflection as a major component of current professional practice and has developed an assessment designed to test this ability. This paper explains why reflection was identified and how this method of assessment was intro- duced and developed. It is itself a reflective account of the process. The context of practice Current professional practice is increasingly complex and diverse. The professional’s role is now clinician, teacher, administrator, consultant and researcher (Shepard & Jensen, 1990). As such demands on professionals change, so, to use Schon’s analogy, does the topography on which they work. Professional development, the means by which professionals enable themselves to fulfil their role, is a life-long process which begins on entering undergraduate study. The problem facing undergraduate education is to facilitate knowledge and skill acquisition (immediate capa- bilities) alongside the development of research skills, through a process of professional clinical reasoning and self-directed learning which will enable professional devel- opment. The immediate capability of new graduates to be able and willing to carry out the responsibilities of pro- fessional practice expected of them soon after graduation, must, in the longer term, be replaced by a responsibility for continuing their own education throughout their pro- fessional life (Feletti, 1984). Facilitation of reflection and the development of reflective abilities is acknowledged to be an important component of this education (Cross, 1993; Day, 1993; Hewson, 1991; Shepard & Jensen, 1990). This process of reflection involves not only consideration of concepts and the execution of skills but also consideration of values and attitudes. These affective aspects are often neglected in professional education because they are intangible and difficult and our knowledge of how learned facts will affect the lives and feelings of others, are aspects of learning that are rarely examined, let alone measured (Ross, 1987). The educational context In the School of Occupational Therapy and Physiotherapy undergraduate education is arranged so that the two pro- fessions follow discrete pathways within a modular course, based on a process model curriculum (Stenhouse, 1975). The process model provides a course which encourages students to become reflective, competent practitioners and to prepare them for a lifetime of continuing learning. This is supported by an active, student lead, self-directed approach to learning. Tutors guide and facilitate students in identifying their learning needs and explore new concepts within the context of their established knowledge and experiences. As the course progresses to the third year, students have opportunities to pursue an individual pathway in optional professional development units reflecting their personal strengths and interest while ensuring core compe- tency. The third year of the course is dedicated to three major concerns: CorrespondLnu: Jenny Routledge, The School of Occupational Therapy and Physiotherapy, University of East Anglia, Nonvich NR4 717, UK. 122 0142-159X/97/020122-07 0 1997 Carfax Publishing Ltd Med Teach Downloaded from informahealthcare.com by Universitat Autonoma Barcelona on 11/04/14 For personal use only.

Upload: richard

Post on 10-Mar-2017

218 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Reflection on the development of a reflective assessment

Medical Teacher, Vol. 19, No. 2, 1997

Reflection on the development of a reflective assessment

JENNY ROUTLEDGE, MOYA WILLSON, MARGARET McARTHUR, BARBARA RICHARDSON & RICHARD STEPHENSON The School of Occupational Therapy and Physiotherapy, University of East Anglia, Nomich, UK

SUMMARY This paper highlights how the School of Occu- pational Therapy and Physiotherapy at the Universig of East Anglia (UEA) has identified reflection as a major component of professional development within the undergraduate curriculum. It describes the evolution, development and implementation of a portfolio-based, reflective method of assessment. Discussion of this experience from both staff and student perspectives identifies several interesting issues. Conrideration of this process points to a professional development of both members of faculty students.

and

In the vaned topography of professional practice, there is a high, hard, ground where practitioners can make effective use of research based theory and techniques, and there is a swampy lowland where situations are confusing ‘messes’ incapable of technical solution. (Schon, 1991, p. 42)

Introduction

Education for the professions must prepare students for the changing world in which they will practice. Assessment should reflect the students’ qualities and achievements which are actively rewarded by the system (Rowntree, 1977). Assessments, therefore, should focus on those abili- ties which are important in the field of practice.

The School of Occupational Therapy and Physiother- apy at the University of East Anglia (UEA) has identified reflection as a major component of current professional practice and has developed an assessment designed to test this ability. This paper explains why reflection was identified and how this method of assessment was intro- duced and developed. It is itself a reflective account of the process.

The context of practice

Current professional practice is increasingly complex and diverse. The professional’s role is now clinician, teacher, administrator, consultant and researcher (Shepard & Jensen, 1990). As such demands on professionals change, so, to use Schon’s analogy, does the topography on which they work. Professional development, the means by which professionals enable themselves to fulfil their role, is a life-long process which begins on entering undergraduate

study. The problem facing undergraduate education is to facilitate knowledge and skill acquisition (immediate capa- bilities) alongside the development of research skills, through a process of professional clinical reasoning and self-directed learning which will enable professional devel- opment. The immediate capability of new graduates to be able and willing to carry out the responsibilities of pro- fessional practice expected of them soon after graduation, must, in the longer term, be replaced by a responsibility for continuing their own education throughout their pro- fessional life (Feletti, 1984).

Facilitation of reflection and the development of reflective abilities is acknowledged to be an important component of this education (Cross, 1993; Day, 1993; Hewson, 1991; Shepard & Jensen, 1990). This process of reflection involves not only consideration of concepts and the execution of skills but also consideration of values and attitudes. These affective aspects are often neglected in professional education because they are intangible and difficult and our knowledge of how learned facts will affect the lives and feelings of others, are aspects of learning that are rarely examined, let alone measured (Ross, 1987).

The educational context

In the School of Occupational Therapy and Physiotherapy undergraduate education is arranged so that the two pro- fessions follow discrete pathways within a modular course, based on a process model curriculum (Stenhouse, 1975). The process model provides a course which encourages students to become reflective, competent practitioners and to prepare them for a lifetime of continuing learning. This is supported by an active, student lead, self-directed approach to learning. Tutors guide and facilitate students in identifying their learning needs and explore new concepts within the context of their established knowledge and experiences.

As the course progresses to the third year, students have opportunities to pursue an individual pathway in optional professional development units reflecting their personal strengths and interest while ensuring core compe- tency. The third year of the course is dedicated to three major concerns:

CorrespondLnu: Jenny Routledge, The School of Occupational Therapy and Physiotherapy, University of East Anglia, Nonvich NR4 717, UK.

122 0142-159X/97/020122-07 0 1997 Carfax Publishing Ltd

Med

Tea

ch D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsita

t Aut

onom

a B

arce

lona

on

11/0

4/14

For

pers

onal

use

onl

y.

Page 2: Reflection on the development of a reflective assessment

Development of a rejlective assessment

independent research activity leading to the submission of a dissertation; client-specific studies leading to the production of sci- entific paper; and a diverse programme of optional professional units com- bined with fieldwork experience sailing under the flag of ‘Professional Development’.

Development of shared learning between the two profes- sions was a key concern in the development of the curricu- lum. This has given us the opportunity to fully examine the needs of a professional in multiprofessional practice and the ways in which both professions work. We have found common acknowledgement of the importance of problem solving, clinical reasoning and reflection and we endeavour to develop these abilities in students from the beginning of the course. Reflection upon the application of the tra- ditional triad of knowledge, skills and attitudes into prac- tice by the student forms the foundation of professional learning. Reflection is both part of the students’ pro- fessional studies and an important part of the ethos of educational practice of the school. We continually evaluate our practice as educators and examine the worth and effectiveness of educational intentions, processes and out- comes and the relationships between them (Adelman & Alexander, 1989). We have come to increasingly value reflection: as a focus for shared learning; as a mechanism for identifymg learning needs and setting directions and goals for moving forward; and as a tool for the develop- ment of the self as a professional within the context of changing health care demands and health care delivery.

Evaluation of a course assessment strategy instigated during the first years of delivery of the course showed that problem solving and clinical reasoning were assessed but not the reflective process. In addition we identified a need for a strategy to bridge undergraduate education and post- graduate professional development. We therefore felt that it was imperative to generate a tool of assessment to satisfy students, staff and professional colleagues of our recogni- tion of the value that we invested in reflection as a key component of life-long professional development.

The debate surrounding the appropriate assessment of ‘Professional Development’ which gave rise to the tool of assessment is described here. In addition to those consider- ations so far presented, a number of practical issues had to be addressed.

When individual students are following different pat- terns of units, and have different clinical experiences, how can one find an equitable form of assessment which is meaningful and not fragmented? Within the process model “assessment is about the teaching of self assessment” (Stenhouse, 1975, p. 95) but a summative result was needed for Honours classification. Since reflective abilities are demonstrated through per- sonal accounts which cannot be anonymized this could create inappropriate anxiety, compromise results or a tendency for students to write more for the assessors than to inform their own learning. The assessment developed should not place a sub-group of students, for example one profession, mature stu-

dents, those with particular pre-course experience or academic bias, at an unfair advantage.

0 Areas of studies such as our Professional Development Module which comprise a number of relatively small modules or pieces of experience present a logistical prob- lem of integration in a single assessment.

0 Whatever tool of assessment emerged it should be owned with confidence by all of those faculty members required to implement it.

Development of the assessment

From the point at which the philosophy and prevailing factors had been informally identified a small working group (which included the authors) took the project for- ward, working over a period of 3 months.

Discussion started with the possibility of the use of a portfolio (see Table 1, paragraph 2) which could be used to present the individual work of students. These have been increasingly given more credence in the documen- tation of experiential and prior learning in continuing professional development in several health-related profes- sions at a post registration stage (Watson & McManus, 1992). Thus a link with postgraduate work could be made.

A learning diary was seen to be an important tool for reflection (Ashbury et al., 1993; Walker, 1989). Glesne & Peshkin (1992) believe that the act of writing itself is rewarding as it ‘creates the product, the housing for the meaing’ i.e. the sense of a professional ‘self’ which all students on professional courses must be striving towards. We see this identity as an essential core element of a professional course which provides a dual qualification of an academic award and a licence to practice. However, if the diary was to be truly reflective and to accurately record the students feelings, observations of colleagues and clients and interactions, it needed to be safe and private. It was, therefore, decided that only student-selected excerpts should be used within the portfolio.

The danger of the portfolio becoming a collection of learning resources was anticipated. We were concerned that the structure of the assessment should reflect the abilities that were being assessed and thus a reflective account would be the distillation of the whole assignment. The portfolio was envisaged as the source of reference for the reflective account and selectivity and appropriateness should, therefore, be required. The portfolio could include items such as the following-key experiences in learning (both formal and experiential); critical incidents; a reflection on these experiences and how this influenced practice; identification of learning sources; evidence of professional growth and learning; anonymous case material. The piece of work would be assessed for the quality of the following items-a reflection on an initial statement of intent in the light of experience; how personal learning, values and theories developed and change; in- sights and understandings gained or altered; implications for future practice; identification of future learning needs and strategies to facilitate meeting these needs.

Thus we arrived at a plan or prototype for the assess- ment but were faced with two dilemmas. First, although the course team had a great deal of educational expertise

123

Med

Tea

ch D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsita

t Aut

onom

a B

arce

lona

on

11/0

4/14

For

pers

onal

use

onl

y.

Page 3: Reflection on the development of a reflective assessment

J. Routkdge et al.

Table 1

PROFESSIONAL DEVELOPMENT Portfolio and Reflective Account

Submission Date:

This assignment has three components

1. Statement of intent This is formatively assessed only (students will be given feedback but no mark will be awarded and it will not count towards the marks for the assessment). Dedicated time will be allowed for this activity. No preparation is required before this day. The assessment will be introduced to the students as a group. Peer discussion will be used to generate ideadguidance for the statement of intent. Students will then generate their individual statements. A supervised room will be allocated to th is task but students may use other rooms available if they so choose. The statement will be submitted by 4.00 pm on that day. This will be read by tutors and students will be given feedback, so that when they return to start the module they will have an early tutorial with the mentor to review the statement of intent. This may be refined and amended, the final version forms the baseline for the assessment.

2. Portfolio During this module and on clinical placement the students will keep a learning diary. The structure and content of this diary are at the discretion of the student, as it is essentially a private document, but f?om its public data must be selected as supportive evidence. Please remember not to use any information which would iden@ any individuals as this would compromise confidentiality. This evidence will be submitted together with the statement of intent as a portfolio. There is no word limit to either the diary or the portfolio but appropriateness, accessibility and clarity will be valued. The diary may include some of the following:

key experiences in learning, both formal and experimental; critical incidents; a reflection on these experiences and how this influenced future practice; identification of learning resources (but not the resources themselves); any evidence of professional growth and learning that is identified; anonymous case material.

3. Reflective account This is a reflective account of the student’s perceived development during th is module. Starting from the statement of intent it will examine the learning experience. Some of these issues may be pertinent:

a reflection on the initial statement of intent in the light of experience; how personal learning, values and theories have developed and changed; insights and understandings gained or altered; implications for future practice; future learning needs and strategies to facilitate these needs.

The resources for this account is primarily the learning diary and students should select and include in the portfolio any dated relevant extracts. The account may be written in the first person and reference to the literature may be made but is not mandatory.

General guidelines apply The reflective account should not exceed 3000 words. There is no word limit to the portfolio.

and enthusiasm there was a need for reassurance in the move away from conventional written assignments and marking guidelines of well established custom and prac- tice. Second, the small working group had travelled a considerable distance in developing the assessment and there was clearly a need to involve the whole course team and ensure common ‘ownership of the assessment’.

Process of implementation of the assessment

Beparation of the staff

The dilemmas were addressed by bringing in an external consultant to work with the small working group in taking

the assessment to a workshop for the whole course team. At the workshop the assessment was presented as a pro- posal for final discussion, modification and refinement. A draft of the assessment together with supporting papers and comments from the external consultant were circu- lated prior to the workshop.

Initially there was anxiety amongst the course team. There was a disbelief and considerable concern that this was not a viable assessment. This may have been indicative of some tension between the technical rational background to professional education and the lack of credit or recogni- tion placed by some on reflection and experiential learning. There were those who were as Schon describes “hungry for

124

Med

Tea

ch D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsita

t Aut

onom

a B

arce

lona

on

11/0

4/14

For

pers

onal

use

onl

y.

Page 4: Reflection on the development of a reflective assessment

Development of a refictive assessment

technical rigor, devoted to an image of solid professional competence, or fearful of entering a world in which they feel they do not know what they are doing” (Schon, 1991, p. 43). They questioned how one could assess what cannot be measured and sought tacit knowledge and specific cri- teria on which to judge the students’ work. The presence of the external expert not only supported and validated the work done by the small working group but also supported the views and concerns of those in the course team who had not been involved in the development. This external validation reassured many of the anxieties about the con- cept of the assessment but left uncertainties about our ability to implement it. A conclusion of the workshop was that each member of the course team would mentor stu- dents in the development of a statement of intent and would act as first marker for the product. This ensured involvement of the whole team so that we would develop together and gain confidence in what we were doing.

Preparation of the students

Students had chosen their ‘Professional Development’ and clinical fieldwork placement options several months before introducing them to this assessment. The programme spans a total of nine teaching weeks arranged either side of an 8-week full-time fieldwork placement. The information that was available to the student at this time is shown in its entirety in Table 1.

A full day was dedicated to the launch of the assess- ment two weeks prior to the commencement of teaching. The reason for this focused introduction was to clarify and expand on the written information and to engage the students in peer group discussion. The assessment was explained and set in the context of the work that they had done so far, the choices that they had made and their future direction and learning as a professional. They were required to produce an initial statement of intent (see Table 1, paragraph 1) and peer group discussion was used for them to generate ideas and reach agreement about what should be included.

Small groups presented their conclusions to the main group and it became clear that the students had a very good perception of what was required and many new ideas which were entirely appropriate. In general the statement of intent included their reasons for choosing their options, how they expected to learn, what they expected to learn, their expectations of where this would lead and how this would relate to future practice. The use of diaries and issues of confidentiality in terms of their own private thought and anonymity of clients and colleagues were discussed. Diaries could include anecdotes; near verbatim accounts of conversations and verbal exchanges; introspec- tive accounts of ones feelings, attitudes, motives, under- standings in reacting to things, events, circumstances; these all help one to reconstruct what it was like at the time (Elliot, 1993). Although such personal biographical writing is necessarily a private activity it was stressed that the students had the freedom to select supportive extracts and to reflect on these in their text. The diary was seen as an essential activity, from which their immedate thoughts could be quoted.

Students were given 4 hours to produce a statement of intent. They had the freedom to do this anywhere and to talk to anyone. This time limit within the launch day lent an element of immediacy and spontaneity to this exercise. The allocation of mentors to students was carried out by the joint directors of the course using the following criteria. First, students were allowed, in confidence, to choose who they did not want as their mentors, this ensured that they would be able to work together constructively. Second, the mentors were to be of the same profession, in order to ensure relevance of advice. Third, the mentors were nei- ther personal tutors nor dissertation supervisors. This en- abled the student to form a fresh relationship in a new context from which mentors then guided the students to refine their statement of intent. The intention was for the mentor to facilitate understanding, to encourage reflection and to comment on the student’s representation rather than put a judgement on it from their own experience of the student.

The refined statement of intent formed a baseline for the portfolio. Quotes from personal recordings and reflections were important parts of the portfolio. We felt that the most important feature of the portfolio should be the selection of extracts which were a resource for the reflective account.

Outcomes

Staff experience

A feature of the staff experience was the many expressions of anxiety. There was concern that they did not know what was expected of them as mentors and a need for reassur- ance that what they were doing was ‘right’. When the pieces of work were handed in there was a heightened level of anxiety among staff which gave rise to a demand for ‘proper guidelines’ and an expression that ‘we don’t know what we are doing’. The small working group had no better idea of what to expect than anyone else and we had our own anxieties and a sense of responsibility for the assessment. What we did have, however, was a belief that the assessment was valid and a hope that we would be able to judge the product. We also felt that we had done as much as we could practically do to prepare the whole course team for the assessment. A considerable amount of informal discussion and sharing of experience took place.

Student experience

Many students expressed similar anxieties to the staff. Many had used learning diaries on placement; the occu- pational therapy students had had more overt encourage- ment to do so. This assessment was very different from any other; they were required to write in the first person and their resource was not literature-based but on their own experience as represented in their diary. They spent 11 weeks on holiday and on clinical placement away from the School and their visiting tutor or tutors were not necess- arily their mentors. Course evaluation questionnaires re- vealed that the students had found the guidelines and process unclear and mentors had been very little help.

However, in the reflective account many students com-

125

Med

Tea

ch D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsita

t Aut

onom

a B

arce

lona

on

11/0

4/14

For

pers

onal

use

onl

y.

Page 5: Reflection on the development of a reflective assessment

J. Routledge et al.

mented on how useful the assessment had been. They had found the experience very challenging but very rewarding.

We are aware that the preparation of the students and the staff both gave rise to clearly expressed anxiety and a demand for more guidance. Students made comments such as ‘the guidelines were vague’, ‘assessment guidelines unclear’ and ‘it was not the writing of the assessment that was difficult, rather the understanding of what was re- quired’. It was also clear that staff did not know what was required and the students commented on this.

Marking experience

All members of the course team marked the assignments of those students that they had mentored. Rather than second marking in the conventional sense, three members of the core team looked at and moderated all scripts. Our con- cern was not only to ensure standardization of marking but to get an overview of what the students had done and how they had tackled the assignment. With all the difficulties of implementing this assessment the result was extremely rewarding. The pieces of work ranged from the largely descriptive to the deeply reflective and analytical. In gen- eral the portfolios were selective and did not outweigh the reflective account. Some students had used the assessment to focus on particular areas of practice while others had taken a broader view. For example, some had looked at therapeutic interventions, others, less specific issues such as their interaction with clients. Many had used the oppor- tunity to reflect on the course as a whole, its philosophy and their experience of it. They were able to see the relevance of some of the course content that they had not previously valued. The accounts of the better students had developed a strategy and direction for future practice. Some exceptional work was done, for example one student had developed her own personal model of professional practice. We had then an unexpected bonus, a rich source of student feedback which was in general very positive.

Marking was based upon the students fulfilment of the requirements of the assessment set out in Table 1. Again the question of how one can assess the unmeasurable arose and staff experienced what students experience, uncer- tainty and apprehension. However, despite the fact that 18 lecturers were involved in marking 55 scripts there was surprising consistency of marking. This contrasted to a very different experience of the same team marking, with very clear guidelines, the same cohort’s dissertations early in the year. The moderators made minor modifications to the marks of a few scripts in consultation with the mentors. Agreement was easily reached; again contrasting to the earlier experience.

One of our overriding concerns from the outset was to ensure that no student fared better or worse than another because of hidher unique learning experience. Although by choosing reflective abilities their particular skills and knowledge were not tested, we suspected that the occu- pational therapy students had had better preparation for reflection than the physiotherapists. On f is t glance this did not appear to be a problem but we subjected the results to statistical testing.

Statistical data

To ascertain whether, for this cohort, there was any suggestion that a particular group of students had been advantaged by the nature of the assignment, the 55 stu- dents (3 male; 52 female) were placed into various sub- groups by reference to information drawn from their UCAS applications.

The sub-groups chosen were: occupational therapy stu- dents; physiotherapy students; students from a predomi- nantly arts background; students from a predominantly science background; school leavers (1 8-20 years); mature (21-30); mature (30 + years); school leavers with a pre- dominantly arts background; school leavers with a pre- dominantly science background. It is here recognized that not all students could be clearly classified by their arts or science background and many were not included in such groups. The ages were determined at the point of entry to the course, not at the time of the assessment. The results spread for each sub-group was described and then each data set that could be compared was done so using the Mann-Whimey U test (minitab software).

Only one test was significant at the 5% confidence level. The sub-group of all students from a predominantly arts background compared to all other students in the cohort scored significantly higher (p = 0.04 16). There was, however, no significant difference between students with predominantly arts or predominantly science backgrounds ($ = 0.1397) or between school leavers from the two back- grounds ($ = 0.0866). School leavers with a predomi- nantly arts background showed a tendency towards higher scores compared with all other students ($ = 0.0543).

The suggestion that arts students fared ‘better’ than non-arts students must be loosely interpreted. Only one cohort is here included and all other sub-group compari- sons were (statistically) insignificant. The trend is here identified for analysis of future cohort results. Indeed, the assessment team take confidence from the data; it appears that the assessment is not truly advantageous to any given group, and age or chosen course (Occupational Therapy or Physiotherapy) do not influence these results.

Our reflections and plans for the future

The results of th is first experience of the assessment are undoubtedly successful. The assessment was seen as valid and fair both from feedback from the students in course evaluation and in the reflective accounts themselves. Most

:YE 12 n l 1 _ 4 2 0

2 30 35 40 45 50 55 60 65 70 75 80 85 Students

N Mean Median TR mean St.Dev. SE mean Min. Max. 01 Q3 55 61.60 58.00 61.63 11.32 1.53 25.00 87.00 55.00 68.00

Figure 1. Results of the whole cohort.

126

Med

Tea

ch D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsita

t Aut

onom

a B

arce

lona

on

11/0

4/14

For

pers

onal

use

onl

y.

Page 6: Reflection on the development of a reflective assessment

Development of a reflective assessment

3.0 2.5 2.0 1.5

4.0 3.5 r

-

- - - - -

n

1.0 - 0.5 -

-

Students N Mean Median TR mean St.Dev. SE mean Min. Max. 0 1 Q3 19 65.21 63.00 66.53 14.20 3.26 25.00 83.00 57.00 79.00

” 45 50 55 60 65 70 75 80 85

Students N Mean Median TR mean St.Dev. SE mean Min. Max. Q1 Q3 36 59.69 57.00 59.09 9.11 1.52 44.00 87.00 55.00 65.00

Figure 2. Results comparison A r t s Background versus Rest. Note: Mann-Whitney Confidence Interval and Test C10 N = 19; Median = 63.00; C17 N = 36; Me- dian = 57.00. Point estimate for ETA1-ETA2 is 6.00. 95.2% CI for ETA1-ETA2 is (0.00, 14.00). W = 647.5. Test of ETAl = ETA2 vs ETAl ne ETA2 is significant at 0.0418. The test is significant at 0.0416 (adjusted for ties).

students found the assessment very valuable. It was appar- ently robust, rigorous and reliable. Assessment of the reflective accounts using the stated criteria showed evi- dence of a move towards independent reflection on experi- ential learning and a commitment to continuing this into their professional lives. There was a feeling of ‘making sense of their course’. One student commented:

After much thought I did finally ‘get to grips’ with what was required and understand it was a

very good assessment in that it made you exam- ine yourself clearly.

It is difficult to identify what else could have been done without any previous experience of any similar assess- ments. Staff clearly lacked confidence in themselves and some were more reluctant than others to enter ‘the swampy lowland’. In the light of our experience of the first run of this assessment, the second run is much easier. We are able to reassure students and staff that this has proved successful and that previous cohort had found it useful. We have been careful also to prepare students more carefully, encouraging the use of learning diaries from the first fieldwork placement. Experience in mentoring and mark- ing has deepened staffs understanding of reflection and they are more confident in mentoring for the second time. During the first year, students and staff were clearly going through a difficult learning progress in understanding how reflection works and how it can be assessed. The danger on future occasions is that in an attempt to reassure students, we may be overprescriptive. We have made one change to the assessment. We initially weighted the marking 30% for the portfolio and 70% for the reflective account, but it became obvious on marking that it was impossible to make a clear distinction between the two since they are so interdependent. A good reflective account was supported by a good selective portfolio. We are now, therefore, assessing it as a whole. We will continue to monitor closely the performance of different groups of students, taking into account the fact that we are constantly developing our curriculum to enhance the student’s reflective abilities.

The process described has been difficult but rewarding. It may be frightening to test one’s beliefs about education but immensely rewarding when the results are not only successful but also appreciated by students. There are very few occasions in professional education when one receives the level of positive feedback contained in these reflective accounts.

Acknowledgements

The authors would like to thank Dr Della Fish for her support in the development of this assessment, and the members of faculty upon whose work and deliberations we draw.

Table 2. Mann-Whimey U test results ~~~

Student sub-group Test result

Occupational Therapy v Physiotherapy A r t s v Science Science v Rest A r t s v Rest School leavers arts v School leavers science School leavers science v Rest School leavers arts v Rest School leavers v Mature (21-30) School leavers v Mature (31 + ) School Leavers v Mature (21 + ) Mature (21-30) v Rest Mature (31 + ) v Rest

p = 0.2960 p = 0.1397 p = 0.6363 p = 0.0416* p = 0.0866 p = 0.3108 p = 0.0543 p = 0.2704 p = 0.5145 p = 0.3135 p = 0.3532 p = 0.6485

* Significant at 0.05.

127

Med

Tea

ch D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsita

t Aut

onom

a B

arce

lona

on

11/0

4/14

For

pers

onal

use

onl

y.

Page 7: Reflection on the development of a reflective assessment

J. Routledge et al.

Notes on contributors

The authors are all Occupational Therapy or Physiotherapy mem- bers of faculty at the University of East Anglia. They have all been involved in the establishment of the new joint school. All have wide experience with education of occupational therapists and physio- therapists in the UK. They were members of the small team which introduced this form of assessment to the school.

References ADELMAN, C. & ALEXANDER, R.J. (1989) The Self-evaluating Znsti-

rution (London, Methuen). ASHBURN, A., BALLINGER, C., BYNG, S., CLAYTON, J., ENDERBY, P.,

MACDONALD, G., PARRY, A., ROBERTS, P.A., ROUTLEDGE, J.,

(1994) Research Development in Occuparional Therapy, Physio- therapy and Speech and Language Therapy: A Posirion Statement (London, Department of Health).

ASHBURY, J.E., FLETCHER, B.M. & BIRIIWHISTLE, R.V. (1993) Personal journal writing in communication skills course for first year medical, Medical Educarion, 27, pp. 196204.

CROSS, V. (1993) Introducing physiotherapy students to the idea of reflective practice, Medical T e a c h , 15(4), pp. 293-307.

DAY, C. (1993) Reflection: a necessary but not sufficient condition for professional development, Brizish Educarion Research Journal,

SKINNER, c., DE SAUZA, L., TEWORD, R. & VAN DER GAAG, A.

19, pp. 83-93.

ELLIOT, J. (1 993) Action Research for Educarional Change (Bucking- ham, Open University Press).

FELE?TI, G.I. (1 984) Assessment for Capability, hgrammed Learn- ing and Educarional Technology, 21(4), pp. 294-300.

GLESNE, C. & PESHKIN, A. (1992) Becoming Qualitative Re- searchers: An Introduction (New York, Longman Publishing Group).

HEWSON, M.G.A.B. (1991) Reflection in Clinical Teaching: an analysis of reflection-on-action and its implications for staffing residents, Medical Teacher, 13(3), pp. 227-231.

Ross, J.M. (1987) Evaluation of Mective Skills, in: Harts Harden (Ed.) Funher Dmelopments in Assessing Clinical Competence (Mon- treal, Can-Heal pub).

ROWNTREE, D. (1977) Assessing Students: How ShaU We Know Them (London, Harpers & Roe).

SCHON, D.A. (1991) The Reflective Practitioner (Avebury, Alder- shot).

SHEPARD, K.F. & JENSEN, G.M. (1990) Physical Therapist Curric- ula for the 1990s: Educating the Reflective Practitioner, Physical Therapy, 70(7), pp. 566575.

STENHOUSE, L. (1975) A n Znmductirm w Curriculum Research and Development (London, Heinemann).

WALKER, M. (1989) Creating a Living Educational Theory from Questions of the kind, ‘How do I Improve my Practice?’, Cam- bridge Journal of EducatMn, 19(1), pp. 41-52.

WATSON, A. & MCMANLJS, M. (1992) hfessional Development Diary (London, The Chartered Society of Physiotherapy).

128

Med

Tea

ch D

ownl

oade

d fr

om in

form

ahea

lthca

re.c

om b

y U

nive

rsita

t Aut

onom

a B

arce

lona

on

11/0

4/14

For

pers

onal

use

onl

y.