online assessment: what influences students to engage with feedback?

284284 © 2014 John Wiley & Sons Ltd. THE CLINICAL TEACHER 2014; 11: 284–289

Feedback

Online assessment: what infl uences students to engage with feedback? Alan Stone, Cardiff University School of Medicine , Cochrane Institute of Primary Care and Public Health , Cardiff , UK

SUMMARY Background : The intention of giving written feedback is to close the gap between the standard achieved and the standard desired, but students do not always read it. Web-based marking tools are increasingly being used in assessment practices to deliver the feedback. What motivates students to read the feedback provided, especially since the advent of these online marking tools, is poorly under-stood. This research looked at the factors likely to infl uence a medical student ’ s engagement with written feedback delivered through an online marking tool ( grademark by Turnitin).

Methods : Third-year medical students on a UK undergraduate medical course submitted an assignment online. A question-naire was distributed to a cohort of them following the release of their results and feedback, allowing quantitative and qualitative data collection. Software recorded whether they opened their feedback. Previous examination performance fi gures were also collated. Results : Online feedback is accessible and acceptable to the majority of students. Personal demographics, computer literacy, previous course performance, or personal motivational drivers did not predict those who did or did

not read it. Some students reported seeing little value in feedback because of their previous negative experiences. A minority found feedback hurtful, and were likely to show avoid-ance behaviours. Discussion: This research found that feedback provided through an online marking tool overcame many of the problems associated with handwritten feedback, but alone was not enough to ensure universal engagement. Feedback dialogues are proposed as a method to overcome negative student experiences, enhance tutor performance and encourage future student engagement.

What motivates students to

read the feedback provided

tct_12158.indd 284tct_12158.indd 284 6/3/2014 12:05:52 PM6/3/2014 12:05:52 PM

© 2014 John Wiley & Sons Ltd. THE CLINICAL TEACHER 2014; 11: 284–289 285

INTRODUCTION

Effective feedback needs to be provided in a meaningful way and must be read, understood

and acted upon by the student. The introduction of online technologies for giving feedback poses new challenges to students, and it is uncertain whether this will result in improved outcomes.

In this medical school, students upload a written assignment during their general practice rotation. Following this, annotated feedback is provided along with a mark. It was noted that the software showed that not all students opened their feedback for long enough to read and make sense of it.

Previous research has shown that most students positively rate formative feedback provided through e-assessment tools. 1 It has also been shown that medical students with poorer marks were less likely than higher performers to collect handwritten feedback. 2 This study assessed whether students opened their online feedback and looked for predictors of who would do so.

The literature identifi es personal demographics, student attitudes towards the course, personal motivational drivers, engagement with the course and previous academic performance as predictors of academic achieve-ment. 3 Computer literacy can affect the ability of a student to access e–learning materials. 4 In addition, a student ’ s previous experience of feedback is also likely to infl uence future behaviours. 5

METHOD

A cohort of students from year 3 of a UK medical course was in-vited to complete a questionnaire after they received their results and feedback on a submitted piece of work. The questionnaire used key questions from tools

that had been validated in other institutions. Free-text comments were also requested for qualita-tive analysis. A manual copy was also posted out so that less tech-nologically competent students were also sampled.

Questions extracted from Vallerand ’ s ‘Academic Motivation Scale’ were used for assessing student motivation. 3 Questions adapted from ‘Students’ Perceptions of Clinical Attachments’ were used for the assessment of the learning environment. 6 The students’ ability to use the information technology tool was assessed using questions adapted from the ‘Student Perception of e–assess-ment Questionnaire’. 7 The sensitivity of the student and utility of the feedback were informed using ‘The Instructional Feedback Orientation Scale’. 5 Students were also asked how long they spent looking at their feedback.

The software recorded whether the students had accessed their feedback for greater than 30 seconds. The software program uses this time period as an indicator that a student has looked at the mark but not allowed a long enough time to adequately read the feedback. The students were ranked into quartiles dependent on their performance in the last set of major examinations. Correlational analysis of the quantitative data was performed using Spearman ’ s rho. Thematic analysis was used to assess the free-text comments of the students.

RESULTS

A high response rate to the questionnaire was achieved, with 81 per cent of the cohort completing the questionnaire. Eighty-eight per cent of all the students sitting this assignment (including non-respondents) opened their feedback for longer

than 30 seconds. All of the stu-dents self-reported reading their feedback. The nine students who did not read their feedback were evenly distributed between the higher and lower academic achievers. All three students who failed their assignment read their feedback. The raw data are presented in Table 1 . Some students failed to answer all questions.

Forty-eight students (64%) thought that they would like to see this type of feedback soft-ware used in the future. Nineteen students (25.3%) found diffi culty in using the software. Fourteen students (18.6%) reported their feelings were easily hurt by corrective feedback, and the same number reported that it increased the stress they felt about future performance.

Four students (5.3%) reported that they could not see the point in looking back to see what they could have done better once they knew that they had passed an assessment.

Analysis of the results showed no correlation between the students’ academic performance, course attendance, strength of motivation, computer literacy, attitude towards the course or attitude to feedback with whether they accessed their feedback.

Post–hoc analysis of the data showed a weak but statistically signifi cant correlation between female gender and those who reported feeling easily hurt by negative feedback (ρ = –0.291, p = 0.011). There was a statisti-cally signifi cant moderate correlation between those students who reported stress from feedback and those not wanting to look back at feedback in the future (ρ = 0.375, p = 0.001).

There were 14 positive comments about the feedback

Effective feedback needs to be provided in a meaningful way



Table 1 . Results ( n = 75) Sex Male 35 (46.7%) Female 40 (53.3%)

Age 20–24 years69(92%)

25–29 years2(2.7%)

>30 years4(5.3%)

Days attended 0 days0

1 day0

2 days0

3 days1(1.3%)

4 days11(14.7%)

5 days63(84%)

Strongly disagree

Disagree Neither agree nor disagree

Agree Strongly agree

I need to show to myself that I can suc-ceed in my studies

0 2 (2.7%) 5(6.7%) 31(41.3%) 36(48.0%)

I feel satisfaction when I accomplish diffi cult academic activities

0 0 1(1.3%) 28(37.3%) 45(60%)

I think I am less hard working than many of the other students in my year

3(4%) 27(36.0%) 20(26.7%) 18(24%) 6(8%)

I experience pleasure and satisfaction while learning new things

0 0 5(6.7%) 47(62.7%) 22(29.3%)

I experience a ‘high’ feeling when read-ing about various interesting subjects

2(2.7%) 3(4%) 20(26.7%) 39(52%) 10(13.3%)

I am ambitious and want to do well in my career

0 1(1.3%) 7(9.3%) 32(42.7%) 33(44%)

I am motivated to work hard to ensure I obtain a more prestigious job later on

1(1.3%) 8(10.7%) 19(25.3%) 28(37.3%) 17(22.7%)

I once had good reasons for studying medicine, now I wonder whether I should continue

37(49.3%) 24(32%) 7(9.3%) 4(5.3%) 1(1.3%)

The honorary lecturer was enthusiastic about teaching

0 0 3(4%) 17(22.7%) 55(73.3%)

I received useful verbal feedback on my progress

2(2.7%) 4(5.3%) 1(1.3%) 30(40%) 38(50.7%)

I got plenty of opportunities to develop my clinical skills

0 0 10(13.3%) 31(41.3%) 34(45.3%)

The honorary lecturer went out of the way to make me feel welcome

0 1(1.3%) 2(2.7%) 22(29.3%) 50(66.7%)

I did NOT understand the ‘hypothesis generation’ method of formulating a differential diagnosis

29(38.7%) 39(52%) 4(5.3%) 2(2.7%) 1(1.3%)

I did NOT understand the importance of the holistic assessment of patients

38(50.7%) 33(44%) 2(2.7%) 1(1.3%) 1(1.3%)

I felt that I was learning what I need to learn to become a competent professional

0 1(1.3%) 4(5.3%) 51(68%) 19(25.2%)

Technical problems caused diffi culty accessing the feedback provided by GradeMark

17(22.7%) 24(32%) 14(18.7%) 15(20%) 4(5.3%)

I found the GradeMark software impractical

13(17.3%) 27(36%) 29(38.7%) 2(2.7%) 3(4%)

The technology used in this online assignment is unreliable

11(14.7%) 31(41.3%) 24(32%) 4(5.3%) 3(4%)

I think the feedback I received in this assignment will help me learn

8(10.7%) 11(14.7%) 9(12%) 38(50.7%) 8(10.7%)

Students with poorer marks

were less likely than higher performers

to collect handwritten

feedback



received on this assignment, particularly the legibility of the feedback, whereas 12 criticised the quality of it.

I could actually under-stand the feedback easily. The problem of unreadable handwriting is resolved! (Student 72)

The grade i [ sic ] re-ceived was good. However, in the places I lost marks there were no comments as to what I could do to improve next time, so I have no idea what I could have done better. (Student 32)

Ten students reported technical issues with the software.

The Document Imager wouldn ’ t work on the university computers. (Student 47)

One student criticised the one-way nature of the feedback.

…there was no way to question any of the com-ments or ask for an ex-panded explanation on some of the points made (Student 11)

Twenty-four students com-plained about their previous negative experience of feedback on the medical course.

Written feedback provided is consistently of a poor standard, often irrelevant and serves no purpose in

terms of improving future performances. (Student 29)

DISCUSSION

The high proportion of students sitting this assignment who opened their feedback confi rms that they mostly found electronic feedback accessible. Their free-text comments suggest that they found this better than handwrit-ten comments. Personal demo-graphics and previous academic performance did not predict who read their online feedback. All of the students who failed the assignment accessed their feed-back. Because of the low numbers failing it would not be possible to categorically state that failure is a predictor of accessing feed-back. There was no correlation between the students’ judgement of their motivational drivers or their personal satisfaction with

In the places I lost marks there were no comments as to what I could do to improve next time

Strongly disagree

Disagree Neither agree nor disagree

Agree Strongly agree

The feedback from GradeMark can add value to my learning

5(6.7%) 8(10.7%) 18(24%) 36(48%) 7(9.3%)

I would like to see this type of feedback used in the future

4(5.3%) 3(4%) 18(24%) 34(45.3%) 14(18.7%)

I think that feedback provides clear direction on how to improve my performance

4(5.3%) 8(10.7%) 19(25.3%) 30(40%) 14(18.7%)

I think feedback from teachers is vitally important in improving my performance

0 0 4(5.3%) 38(50.7%) 33(44%)

My feelings can be easily hurt by corrective feedback from a teacher/tutor

12(16%) 30(40%) 19(25.3%) 10(13.3%) 4(5.3%)

Corrective feedback from a teacher increases the stress I feel about future performance

10(13.3%) 37(49.3%) 12(16%) 12(16%) 2(2.7%)

Once I know I have passed an assignment I can ’ t see the point in looking back to see what I could have done better

19(25.3%) 39(52%) 13(17.3%) 3(4%) 1(1.3%)

How long did you spend studying your written feedback?

I didn ’ t read the feedback

Less than 5 minutes

5–10 minutes

More than 10 minutes

How long did you spend studying your written feedback?

0 33(44%) 32(42.7%) 10(13.3%)

Table 1. continued



this module with whether they accessed their feedback.

A minority of students had technical issues with the soft-ware, but most were able to overcome these problems; an even smaller minority thought that the software was impracti-cal. These results show that medical students are mostly computer literate, but that a minority found the technology challenging. There was no correlation between the students who reported diffi culties access-ing the feedback and those who failed to read it. This would suggest that there were factors other than technical issues that ultimately inhibited access.

The large number of students who did not feel that feedback was likely to improve their performance was compatible with the low rating of feedback in the National Student Survey. Most of these medical students agreed that feedback was vital for learning, but many reported that in practice it often did not achieve this goal. A minority of respondents reported that their feelings could be hurt by feed-back and that feedback could increase their stress about future performance. An important fi nding of this research is that students who stated that their feelings were easily hurt were more likely to report that they could see little value in looking back at feedback once they had passed an assessment.

There are wide discrepancies between lecturers and students in their perceptions of how effec-tive feedback is: most lecturers think their feedback helps understanding and learning, whereas students often disagree. 8 It has been previously argued that feedback should encourage an exchange between the teacher and the student in order to maximise student understanding. 9 It is proposed that in future, written feedback should only be

seen as the initiation of a post-performance discussion between student and tutor. Electronic methods of communi-cation, such as e–mail, could facilitate a cost-effective use of the tutor ’ s time.

If written feedback continues to be seen as a one-way fl ow of information from tutor to student, then it is unlikely that the tutor will learn what the student requires and whether the statements made will be under-stood. In addition, sensitive students may not perceive the feedback in the way that it was intended. In the absence of the tutor receiving feedback from the student, it is likely that the tutor will continue to make the same mistakes in the future. 10 By engaging in a feedback discourse, it is more likely that tutors will learn to be sensitive to student feelings and avoid the comments that are perceived as being hurtful (Figure 1 ).

Inevitably, some students will see a dialogue with the marker as an opportunity to question the grade that they received. It should be possible to prevent the discourse developing at cross-purposes if it is made clear in advance that the reason for the discourse is to look forwards to improve future performance, and not to alter grades. A feedback

dialogue provided through online media is more likely to leave students feeling engaged and address their learning needs in a student-centred learning intervention.

REFERENCES

1 . Stevens K , Jamieson R . The introduction and assessment of three teaching tools (WebCT, Mindtrail, EVE) into a Post-graduate course . Journal of Information Technology Education 2002 ; 1 : 233 – 252 .

2 . Sinclair HK , Cleland JA . Undergraduate medical students: who seeks formative feedback? Med Educ 2007 ; 41 : 580 – 582 .

3 . Vallerand R , Pelletier L , Blais M , Briere N , Senecal C , Vallieres E . The Academic Motivation Scale: A meas-ure of intrinsic, extrinsic, and amo-tivation in education . Educational and Psychological Measurement 1992 ; 52 : 1003 – 1017 .

4 . Link TM , Marz R . Computer literacy and attitudes towards e– learning among fi rst year medical stu-dents . BMC Medical Education 2006 ; 6 : 34 .

5 . King P , Schrodt P , Weisel J . The Instructional Feedback Orientation Scale: Conceptualizing and Validating a New Measure for Assessing Perceptions of Instructional Feedback . Communication Education 2009 ; 58 : 235 – 261 .

6 . Lyon PM , McLean R , Hyde S , Hendry S . Students’ perceptions of clinical attachments across rural and met-ropolitan settings . Assessment

Lecturers think their feedback

helps understanding and learning,

whereas students often

disagree

Figure 1 . Proposed outline of a student–tutor dialogue



& Evaluation in Higher Education 2008 ; 33 : 63 – 73 .

7 . Dermo J . e–Assessment and the student learning experience: A survey of student perceptions of e–assess-ment . British Journal of Educational Technology 2000 ; 40 : 203 – 214 .

8 . Carless D . Differing percep-tions in the feedback process . Studies in Higher Education 2006 ; 31 : 219 – 233 .

9 . Rae AM , Cochrane DK . Listening to students: How to make written assessment feedback useful .

Active Learning in Higher Education 2008 ; 9 : 217 – 230 .

10 . Hattie JAC , Timperley H . The power of feedback . Review of Educational Research 2007 ; 77 : 104 .

Corresponding author ’ s contact details: Dr Alan Stone, Cardiff University School of Medicine, Cochrane Institute of Primary Care and Public Health, Neuadd Meirionydd, Heath Park Campus, Cardiff, CF14 4YS, UK. E-mail: [email protected]

Funding: This research was funded as part of my MSc project by the Cochrane Institute of Primary Care and Public Health, Cardiff University.

Confl ict of interest: None.

Ethical approval: Approval granted by Postgraduate Medical and Dental Education Research Ethics Committee Cardiff University School of Medicine.

doi: 10.1111/tct.12158

Feedback dialogue provided through online media is more likely to leave students feeling engaged


online assessment: what influences students to engage with feedback?

Documents